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Simulation  Based  Acquisition  is  a 
term  used  in  DoD  and  the  Defense 
Industry  recently  to  convey  the  idea 
of  actively  incorporating  simulation  tech¬ 
nology  into  acquisition  processes,  extending 
that  integration  beyond  the  traditional 
stovepipes  within  the  acquisition  communi¬ 
ty,  and  further  extending  to  the  require¬ 
ments,  training  and  logistics  communities. 
The  concept  conveys  that  the  sharing  of  the 
developed  models  and  simulations  will  be 
made  across  acquisition  programs  and 
across  the  Services  in  some  cases.  The  con¬ 
cept  also  implicitly  demands  a  robust  con¬ 
figuration  control  process  to  ensure  the  sim¬ 
ulation  at  all  times  represents  the  intended 
product.  The  DoD  vision,  as  stated  by  Dr. 
Jacques  Gansler,  Under  Secretary  of 
Defense  for  Acquisition  and  Technology, 
and  Dr.  Patricia  Sanders,  Director,  Test, 
Systems  Engineering  and  Evaluation  for 
OSD,  is  that  Simulation  Based  Acquisition 
will  generate  savings  in  cost  and  schedule, 
as  well  as  reducing  risk. 

These  thoughts  become  more  important 
in  the  environment  of  decreased  funding 
and  manpower  when  we  are  making  efforts 
to  get  advanced  weapons  to  the  field  soon¬ 
er. 

The  challenge  is  to  get  stakeholders  to 
commit  to  this  process  by  developing  robust 
models  and  simulations,  ^d  using  them  in 
a  collaborative  fashion.  The  step  to  be 
taken,  beyond  what  a  weapon  system  pro¬ 
gram  using  modeling  and  simulation  in 
acquisition  does  now,  is  to  develop  simula¬ 
tion  technologies  for  more  than  one  func¬ 
tion  (e.g.  training  or  testing)  or  for  multiple 
acquisition  programs. 
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The  intent  of  Simulation  Based  Acquisi¬ 
tion  for  the  Army  is  to  use  simulation  tech¬ 
nology  to: 

1.  Explore  the  trade  space  when  identify¬ 
ing  performance  requirements; 

2.  Assess  design  alternatives  for  perfor¬ 
mance,  logistics,  manufacturability, 
operability,  maintainability,  cost,  etc.; 
and, 

3.  Provide  soldiers  with  training  oppor¬ 
tunities  without  leaving  the  motor 
pool  parking  lot. 

Cradle  to  Grave.  The  question 
becomes  how  to  implement  this  idea  of 
breaking  out  of  the  traditional  stovepipes  in 


which  we’ve  done  business.  The  models 
developed  during  concept  development  by 
the  Army’s  Training  and  Doctrine  Com¬ 
mand  can  be  incorporated  into  work  done 
by  the  requirements  technical  community. 
Virtual  Prototypes  can  be  used  to  do  some 
of  the  groundwork  that  was  previously  done 
by  building  physical  prototypes,  for  exam¬ 
ple;  design  level  for  stabihty,  etc.  But  that 
is  not  enough,  there  are  still  tests  that  need 
to  be  run  at  some  point.  What  the  simula¬ 
tions  and  prototypes  can  do  is  allow  the 
Operational  Tester  to  focus  on  what  really 
needs  to  be  tested  in  the  field.  The  Army 
Acquisition  Community  captures  most  of 
this  in  what  they  call  ‘SMART,’  Simulation 
and  Modeling  for  Acquisition,  Require¬ 
ments  and  Training.  They’re  on  the  right 
path  but  we  still  need  to  work  the  logistics 
tail  into  what  we’re  doing  to  achieve  Simu¬ 
lation  Based  Acquisition. 

Simulation  technology  becomes  an 
enabler  to  work  the  interfaces  between 
requirements  and  acquisition,  as  well  as 
acquisition,  training  and  logistics.  In  a  very 
real  sense,  the  acquisition  community  is 
somewhat  held  hostage  by  the  requirements 
community  when  a  ‘waterfall’  approach  is 
used,  just  as  the  training  community  is  held 
hostage  to  the  decisions  made  by  the  acqui¬ 
sition  community.  Using  an  environment 
of  linked  simulations  we  can  accomplish 
more  parallel  efforts  instead  of  the  handoffs 
of  the  serial  approach. 

Challenges.  One  of  the  challenges  to 
increased  use  of  simulation  technology  is 
who  maintains  the  models  and  keeps  them 
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MORS  PRESIDENT 


Developing  the  Junior  Analyst  for  1998/99  —  Halftime  Report 


By  the  time 
you  read  this 
article,  the 
current  MORS  team 
will  be  almost 
halfway  through  their 
term  of  office.  I  will 
briefly  talk  about 
accomplishments  to 
date  and  discuss  our 
strategy  for  the  sec¬ 
ond  half  of  the  ballgame.  A  balance 
between  the  meeting  operations  and  the 
professional  affairs  side  of  the  house  has 
been  maintained.  A  quick  review  of  the 
first  half  is  provided  at  the  beginning  of 
each  section,  with  the  necessary  focus  for 
the  second  half  following. 

Meeting  Ops 

With  Dr.  Roy  Rice  (Teledyne  Brown 
Engineering)  at  the  overall  helm  of  Meet¬ 
ing  Ops,  we  have  successfully  completed 
two  special  meetings.  Dr.  Cy  Staniec 
(Logicon),  our  Special  Meetings  Chair,  has 
provided  the  leadership  at  the  detail  level 
for  both  past  and  present  meetings  this 
year.  The  first  meeting,  held  in  August, 
helped  focus  the  Simulation  Technology 
ten  years  ahead.  Dr.  Stu  Starr  did  a 
superb  job  of  leading  this  three  day  effort 
and  a  summary  of  the  meeting  will  be  pro¬ 
vided  in  a  future  issue?  MORS  veiy  much 
appreciates  the  level  of  effort  provided  by 


Upcoming  MORS 
Meetings 

SIMVAL99 

26, 27, 28  January  1999 

The  Johns  Hopkins  University/APL 

Laurel  MD 

Joint  Experimentation  Workshop 

8, 9, 10, 11  March  1999 

67th  MORS  Symposium 

22, 23, 24  June  1999 
US  Military  Academy 
West  Point,  NY 


RADM  Nutwell  in  the  C4ISR  Workshop 
held  at  the  Army  War  College.  We  also 
appreciate  the  efforts  of  CAPT  Jay  Kistler 
(N6M),  Deputy  Chair;  Dr.  Russ  Richards 
(Mitre),  Technical  Chair;  and  both  Pat 
Peterson  and  Barry  Dysart  (SAIC)  for 
their  efforts  in  organizing  this  great  work¬ 
shop.  Details  of  this  workshop  will  be  pro¬ 
vided  in  the  next  issue  of  PHALANX.  Ms. 
Anne  Patenaude  (SAIC)  has  done  a  great 
job  to  date  with  her  “Brown  Bag”  lunches 
guiding  the  rest  of  her  67th  staff  in  prepar¬ 
ing  for  the  67th  MORSS  this  summer  at 
West  Point.  A  word  to  the  wise  for  every¬ 
one,  hotels  are  at  a  premium,  so  make 
reservations  early.  CDR  Kirk  Michealson, 
(OSD  PA&E),  has  done  a  super  job  in 
organizing  the  Working/Composite  Groups 
this  year.  His  monthly  newsletters  and 
“enormous”  flow  of  e-mail  has  increased 
the  communications  between  all  33  work¬ 
ing  groups  this  year.  I  firmly  believe  the 
Working  Groups/Composite  Groups 
(WG/CG)  are  basically  the  heart  and  soul 
of  our  Society,  and  they  should  be  used 
year  around,  not  only  at  our  annual  sympo¬ 
sium. 

Second  Half  Strategy  includes  the  suc¬ 
cessful  completion  of  both  the  SIMVAL 
99  at  JHU/APL  and  the  Joint  Experimenta¬ 
tion  Workshops  in  the  Norfolk,  VA  area. 
The  WG/CG  chairs  and  the  67th  Staff  will 
continue  to  build  a  strong  program  for  the 
West  Point  summer  event.  We  will  also 
start  looking  at  potential  Special  Meetings 
for  next  fall. 

Professional  Affairs 

CAPT  Lee  Dick’s  (N75)  positive  lead¬ 
ership  has  helped  keep  the  Professional 
side  of  our  house  in  good  order.  Publica¬ 
tions  of  the  Society  are  being  brought  on- 
track  by  Mr.  Brian  Engler  (Systems  Plan¬ 
ning  &  Analysis).  The  backlog  of  Special 
Meeting  reports  will  almost  be  complete 
by  the  time  this  article  is  published.  These 
reports  are  a  true  test  of  the  Society’s  lead¬ 
ership  and  ability  of  a  volunteer  society  to 
provide  a  product  away  from  the  MORS 
work  environment.  I  would  appreciate 
everyone’s  effort  in  completing  their  part 


of  the  report  before  they  leave  any  work¬ 
shop.  MAJ  Willie  McFadden  (PhD  stu¬ 
dent,  Old  Dominion  University)  has 
already  began  making  plans  for  the  Spring 
Education  Colloquium,  to  be  held  in  the 
D.C.  area.  Stay  tuned  on  the  MORS  web 
site  for  further  information.  Mr.  Ted 
Smyth  (JHU/APL)  is  overseeing  a  Opera¬ 
tions  Research  oral  history  project  by  Mr. 
Gene  Visco.  Maj  Mark  Gallagher  has 
begun  to  coordinate  the  reading  of  both  the 
Rist  and  Barchi  Prize  papers.  The  winners 
will  be  announced  during  one  of  the  Spe¬ 
cial  Sessions  at  the  67th  MORSS. 

Second  half  focus  will  be  on  the  Spring 
Education  Colloquium  to  support  the 
MORS  theme  of  the  Junior  analyst. 

The  great  efforts  of  finance  and  man¬ 
agement  led  by  Ms.  Sue  Iwanski  and  Sec¬ 
retary  of  the  Society,  Dr.  Tom  Allen  will 
be  discussed  in  the  next  issue. 

Use  of  the  Theme  - 
“Developing  the  Junior  Analyst” 

We  hope  everyone  gets  on  board  with 
this  theme.  The  MORS  Executive  Council 
has  taken  the  lead  by  each  adopting  a 
junior  analyst  to  mentor  during  the  year. 
Time  is  spent  weekly  by  each  senior  ana¬ 
lyst  providing  analytical  knowledge  and 
methodologies  not  normally  obtained  in 
any  textbook.  Other  committees  have 
ensured  this  theme  is  presented.  The 
WG/CGs  have  promoted  the  Junior  Ana¬ 
lyst  as  one  of  the  WG  co-chairs  during  the 
67th  MORSS.  We  are  planning  on  having 
two  Junior/Senior  Analysts  sessions  this 
year. 

I  made  a  delightful  trip  to  the  United 
States  Air  Force  Academy  in  October  as  a 
guest  speaker  for  the  Junior  and  Senior 
Seminars.  I  discussed  the  basics  of  analy¬ 
sis,  what’s  going  on  in  D.C.  area  analytical 
scene  and  gave  a  quick  introduction  to 
MORS.  I  also  spent  time  with  two  cadets 
discussing  their  summer  research  projects. 
It  is  certainly  great  knowing  the  Operation 
Research  community  has  such  a  bright 
future  with  the  ladies  and  gentlemen  gradu¬ 
ating  soon. 

(See  MORS  PRESIDENT,  p.  32) 
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MAS  —  The  Next  Phase 


Tom  Gulledge 

MAS  President 


y  the  time  this 
column  is 
published,  I 
will  no  longer  be  the 
President  of  MAS. 
The  previous  two 
years  have  been  pro¬ 
ductive,  and  I  am 
pleased  to  leave  a 
financially  viable  and 
growing  organization 
to  my  successor,  Dr.  Bruce  Fowler 
[FOWLER-BW@redstone.army.mil] . 
Bruce  and  I  are  in  close  contact  (almost 
daily),  and  the  transition  is  already  under¬ 
way.  Bruce  has  some  great  ideas  and  a 
good  vision  for  the  future  of  MAS  —  your 
Society  is  in  good  hands. 

The  transition  of  officers  will  take  place 
at  the  Seattle  INFORMS  meeting,  and 
Bruce  will  report  the  results  of  the  election 
in  his  first  PHALANX  column.  We  are 
planning  a  large  mihtary  meeting  for  Seat¬ 
tle,  as  well  as  a  management  off-site  to 
plan  for  the  future  of  MAS.  A  number  of 
change  initiatives  are  underway  in 
INFORMS,  including  a  move  to  a  single 
annual  national  meeting.  When  this  transi¬ 
tion  is  completed,  it  is  almost  certain  that 
our  MAS  Council  will  propose  an  annual 
MAS  meeting.  In  fact,  the  second  MAS 
National  Conference  is  already  planned  for 
next  fall  in  the  southwest.  The  details  will 
follow  later,  but  here  is  some  advance 
information. 

The  US  Army  Air  Defense  Center  and 
TRAC  White  Sands  will  co-host  the  Sec¬ 
ond  International  Military  Applications 
Society  Conference  during  the  last  week  of 
September  1999.  The  conference  chair  will 
be  Philipp  Djang  [djang@trac.wsmr. 
army.mil].  Thanks  to  the  superior  facilities 
and  support  accorded  by  the  US  Army  Air 
Defense  Center,  you  can  expect  a  first  class 
conference.  However,  this  major  event  of 
the  society  cannot  take  place  without  your 
support  and  personal  efforts.  We  would 
like  to  encourage  volunteers  for  track 
chairs.  Some  of  the  tracks  we  propose  are: 

•  Theater  Missile  Defense 

•  Space  Operations 

•  Electronic  Warfare 


•  Joint  Campaign  Analysis 

•  Joint  Mobihty  Analysis 

•  Naval  Operations  Analysis 

•  Logistics,  Rehability  and 
Maintainabihty 

•  Modehng  Low  Intensity  Conflict  and 
Operation  Other  Than  War 

•  Advanced  Warfighting 
Experimentation 

•  Future  Impacts  of  Budgetary 
Reductions  on  Force  Design 

•  Manpower  Modeling 

•  Cost  and  Resource  Analysis 

•  Readiness,  Personnel,  Training 

•  Distributed  Simulation  and  Distributed 
Testing 

•  Military  Decision  Analysis 

•  Advanced  Operations  Research 
Methods 

•  C4IR  and  Intelligence  Operational 
Analysis 

Please  contact  Philipp  for  conference 
information,  or  to  volunteer  to  participate 
as  a  track  or  session  chair.  A  call  for 
papers  will  be  issued  shortly. 

The  second  featured  item  of  this  col¬ 
umn  is  our  continued  focus  on  the  MOR 
programs  in  our  service  academies.  If 
there  is  one  area  that  has  become  closely 
associated  with  my  presidency,  it  has  been 
the  focus  on  our  MOR  educational  institu¬ 
tions.  In  the  last  PHALANX  issue,  I  report¬ 
ed  on  the  award  that  was  presented  at  the 
US  Military  Academy.  In  this  issue,  I 


Thomas  Gulledge  presenting  the  MAS 
Award  to  Cadet  Scott  D.  McKeever 


focus  on  the  US  Air  Force  Academy. 

This  year  the  Air  Force  Academy  grad¬ 
uated  69  cadets  in  OR,  the  largest  class  to 
date.  The  recipient  of  the  MAS  award  for 
the  best  cadet  in  OR  was  Cadet  Scott  D. 
McKeever.  Cadet  McKeever  received  a 
Draper  Fellowship  in  Operations  Research 
and  will  continue  his  studies  at  MIT  this 
fall.  Following  his  Master’s  Degree,  he 
will  attend  the  premier  undergraduate  pilot 
program,  Euro-NATO,  at  Sheppard  AFB, 
Texas. 

Cadet  McKeever  completed  his  OR 
program  by  serving  as  cadet  team  leader  of 
a  call  center  simulation  project  for  USAA, 
Inc.  The  USAA  Rocky  Mountain  Region 
is  expanding  their  operations  by  2  hours 
daily.  The  Regional  Vice  President  per¬ 
sonally  attended  the  cadets’  final  briefing 
and  was  impressed  with  their  analysis.  The 
model  is  being  used  by  USAA  analysts  to 
simulate  the  effects  of  various  staffing  pro¬ 
files  of  the  extended  hours  of  operation. 
Congratulations  to  Cadet  McKeever  for  his 
achievements  and  we  wish  him  the  best  as 
he  continues  his  academic  career  at  MIT. 

As  usual,  feel  free  to  contact  me  by 
Internet  [gulledge@gmu.edu].  The  MAS 
Council  continually  monitors  the  MAS 
Listserver,  and  you  can  always  contact  the 
membership  or  us  by  this  means.  If  you 
are  not  already  a  member  of  the  list,  you 
can  subscribe  by  sending  the  following 
message  to  majordomo@mat.gsia. 
cmu.edu: 

subscribe  mas  Your  Name, 

Title  <  youmame@domain.org> 

If  you  have  problems,  send  a  note  to 
Phihpp  Djang.  He  doubles  as  the  modera¬ 
tor  of  the  list.  I  will  use  the  listserver  to 
provide  information  of  interest  to  the  Mili¬ 
tary  Operations  Research  community  as  it 
is  passed  to  me.  I  encourage  you  to  do  the 
same. 

Finally,  I  thank  you  for  all  of  your  sup¬ 
port  during  the  last  two  years.  Military 
Operations  Research  is  alive  and  well,  and 
through  your  continued  support  of  organi¬ 
zations  like  MAS,  we  will  continue  to 
grow  our  profession.® 
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Supporting  the  Decision  Makers 


Dr.  Tom  Allen 
IDA 


Customarily, 
officers  of 
MORS  are 
given  one  opportu¬ 
nity  each  year  to 
discuss  new  devel¬ 
opments  in  their 
area,  so  that  readers 
can  keep  up  with 
the  many  advances 
going  on  through¬ 
out  the  Society.  Fortunately  for  me, 
two  committees,  with  superb  track 
records  and  very  able  leadership,  sup¬ 
port  the  Secretary:  Membership  under 
Dean  Hartley,  and  Electronic  Media, 
led  by  Glen  Johnson.  Stable  member¬ 
ship  size  despite  major  reductions  in 
defense,  and  effective  communication 
about  MORS  and  its  events  resulting 
from  a  constantly  improving  web  site, 
underline  the  contributions  of  these 
groups.  Suffice  it  to  say  that  both  com¬ 
mittees  welcome  suggestions  and  vol¬ 
unteer  help  to  maintain  this  record  of 
success. 

Instead  of  discussing  secretarial  mat¬ 
ters,  I’d  like  to  address  another  area  of 
importance  to  MORS,  which  is  the  rela¬ 
tionship  of  analysts  to  the  decision 
makers  and  commanders  they  support. 
Thirty  years  of  experience  in  the  busi¬ 
ness  listening  to  comments  like  those 
provided  by  BG  John  Scales  in  his  arti¬ 
cle  in  the  last  edition  of  PHALANX 
have  convinced  me  that  our  community 
has  not  been  ineffective  in  convincing 
defense  leaders  about  the  full  value  of 
our  profession.^ 

There  is  no  question  that  Secretary 
of  Defense  MacNamara  brought  a  new 
focus  of  cost  effectiveness  to  military 
operations  research,  with  new  systems 
undergoing  detailed  scrutiny  and  hav¬ 
ing  to  prove  their  worth  by  demonstrat¬ 
ing,  at  least  on  paper,  improved  perfor¬ 
mance  for  less  cost  than  alternatives. 
This  fueled  the  development  of  a  vast 
array  of  new  models,  methodologies 
and  approaches  to  enable  manufactur¬ 
ers,  acquisition  agents,  commanders 
and  high-level  decision  makers  to  com¬ 


pare  the  effectiveness  and  cost  of  vari¬ 
ous  proposed  systems  in  battlefield  con¬ 
ditions.  Analysts  have  responded  to 
this  environment  by  reviewing  various 
options  of  differing  force  size  and 
equipment  in  order  to  help  decision 
makers  determine  which  of  the  alterna¬ 
tives  meets  the  stated  requirements  at 
least  cost. 

While  this  is  an  important  aspect  of 
analysis,  it  is  far  from  being  the  com¬ 
munity’s  only  contribution.  We  go 
astray  whenever  we  allow  our  focus  to 
be  reduced  to  just  the  engineering  val¬ 
ues  and  cost  factors  associated  with  a 
comparison  of  new  systems.  The  prob¬ 
lem  is  that  even  when  we  know  the 
issues  are  more  complex,  we  often 
assume  away  critical  variables  (espe¬ 
cially  human-related  factors)  because 
we  and  the  commanders  we  serve  do 
not  have  quantifiable  measures  in  hand 
to  describe  the  effect  of  these  variables 
on  the  battlefield. 

The  truth  is,  military  operations 
research  has  much  to  offer  in  address¬ 
ing  these  complex  variables.  We  need 
to  work  with  operational  experts  to  both 
define  them  and  better  understand  their 
effects.  We’ve  solved  most  of  the  easy 
problems;  now  we  need  to  understand 
the  impact  of  human-related  variables 
such  as  tactical  execution,  readiness, 
morale  and  training  on  battlefield  out¬ 
come  and  then  include  the  appropriate 
measures  in  our  analyses. 

Studying  these  factors  is  not  just  the 
purview  of  leadership  and  military  sci¬ 
ence;  operations  researchers  are  oblig¬ 
ated  to  bring  our  scientific  approaches 
to  bear  and  assist  in  this  study.  It  is  a 
fact  that  the  best  commanders  look  for 
ways  to  understand  and  apply  these 
human-related  factors  to  their  advan¬ 
tage  on  the  battlefield.  The  fact  that 
such  variables  are  hard  to  define  and 
measure  does  not  mean  they  should  be 
ignored.  In  fact,  as  Napoleon  and  Gen¬ 
eral  Scales  point  out,  the  impact  of 
these  variables  can  far  exceed  any  mar¬ 
ginal  improvement  to  firepower  associ¬ 
ated  with  fielding  new  systems  or  force 


structures. 

This  does  not  mean  that  the  OR 
community  should  stop  looking  at  effi¬ 
ciency  and  effectiveness.  Far  from  it, 
with  the  continuing  pressure  on  military 
budgets,  the  services  and  the  Depart¬ 
ment  of  Defense  must  find  least  cost 
solutions  to  an  ever-increasing  number 
of  modernization  and  force  structure 
challenges.  What  it  does  mean  is  that 
the  OR  community  must  also  expand  in 
other  dimensions  to  meet  the  full 
demands  of  the  customers  they  serve. 

As  in  our  other  efforts,  we  must  cre¬ 
ate  the  databases,  develop  the  theories 
and  then  shape  the  tools,  methods  and 
approaches  necessary  to  help  comman¬ 
ders  deal  with  the  full  range  of  opera¬ 
tional  issues.  Analysts  need  to  provide 
insight  into  the  effects  of  training  as 
well  as  to  expand  the  community’s 
understanding  of  strategic  impacts  and 
maneuver-based  warfare  in  order  to 
help  shape  battlefield  options  to  achieve 
victory  more  quickly  and  decisively 
than  traditional  attrition-based  ap¬ 
proaches. 

General  Scales  is  right  when  he 
points  out,  “...research  analysts  are 
often  not  at  the  right  hand  of  the  deci¬ 
sion  maker. The  reason  is  not 
because  the  analytic  community  is  inca¬ 
pable  of  assisting  commanders,  but 
rather  because  we  analysts  have  not 
shown  the  commanders  that  we  can 
bring  relevant  thought  to  bear  on  their 
toughest  problems. 

No  commander  worth  his  or  her  salt 
wants  to  be  dictated  to  by  an  opaque 
computer  model  that  everyone  knows  is 
missing  relevant  information;  on  the 
other  hand,  every  good  commander  I’ve 
ever  served  is  eager  to  listen  to  smart, 
objective  insights  to  help  improve  their 
understanding  of  a  problem.  The  more 
analysts  provide  such  relevant  insight 
—  improving  both  our  own  and  the 
decision  makers  grasp  of  an  issue  and 
ability  to  make  credible,  defendable 
choices  —  the  more  likely  we  will  find 
ourselves  invited  into  the  decision-mak- 
VEEPS  PEEP,  p.  29) 
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Information  Superiority  and  Game  Theory: 
The  Value  of  Information  in  Four  Games 


Dr.  Jerome 
Bracken 
RAND 

This  article  is  based  on  work  performed 
in  the  RAND  Arroyo  Center  for  the  US 
Army. 

OVERVIEW 

hree  overarching  concepts  tend  to 
frame  predictions  about  the  future 
in  which  US  military  forces  are 
expected  to  operate.  The  first  of  these  is 
the  notion  that  an  Information  Age  is  just 
beginning  to  unfold  and  that  it  will  largely 
define  the  first  half  of  the  21 -century.  Like 
the  Industrial  Revolution  that  held  sway 
throughout  the  latter  half  of  the  19th  centu¬ 
ry,  the  new  era  promises,  among  other 
things,  to  transform  the  nature  of  future 
military  operations  —  so  much  so  that  it  is 
predicted  to  result  in  a  Revolution  in  Mili¬ 
tary  Affairs  (RMA). 

Such  a  revolution  is  the  second  major 
concept  currently  framing  predictions 
about  future  forces.  For  an  RMA  to  occur, 
the  role  of  information  —  its  technologies 
and  their  organization  —  is  generally  con¬ 
sidered  to  be  critical.  In  particular,  through 
the  development  of  Information- Age  tech¬ 
nologies,  the  RMA  is  expected  to  produce 
Information  Superiority  (the  third  overar¬ 
ching  concept),  which  future  US  forces  are 
expected  to  enjoy  over  their  opponents.  A 
“vision”  of  information  superiority,  in  fact, 
pervades  Joint  Vision  2010,  where  such 
superiority  is  defined  as  the  “capability  to 
collect,  process,  and  disseminate  an  unin¬ 
terrupted  flow  of  information  while 
exploiting  or  denying  an  adversary’s  abih- 
ty  to  do  the  same.”^ 

We  focus  here  on  this  notion  of  infor¬ 
mation  superiority.  We  want  to  explore  its 
conceptual  dimensions  somewhat  further. 
In  particular,  we  would  like  to  understand 


how  much  information  superiority  might 
be  required  —  how  much  is  enough,  if 
you  will  —  for  US  military  forces  to  be 
able  to  enjoy  truly  revolutionary  advan¬ 
tages  over  their  opponents  in  an  Informa¬ 
tion-Age  future. 


To  address  the  question  of  how  much 
information  might  be  required  for  US 
forces  to  achieve  superiority,  we  have 
drawn  upon  game  theory  as  a  methodolo¬ 
gy  that  looks  to  be  directly  relevant  to 
such  questions.  In  the  applications  of 
game  theory  presented  below,  therefore, 
we  have  compared  military  operations  to 
a  zero-sum,  two-sided  game.  For  these 
games,  we  posited  that  each  side  has  3,  5, 
or  10  choices  or  “strategies”  available  for 
achieving  victory. 

For  each  game,  we  vary  the  kinds  of 
information  available  to  the  two  sides. 
The  3x3,  5x5,  and  10x10  rows  in  the  chart 
above,  therefore,  could  be  taken  to  repre¬ 
sent  the  probabilities  of  victory  or  “pay¬ 
offs”  for  Side  1  (the  US).  In  fact,  these 
payoffs  represent  the  calculated  averages 
of  1000  trials  of  3x3,  5x5,  and  10x10 
games  composed  of  random  numbers  uni¬ 
formly  distributed  between  0  and  100.  A 


detailed  description  of  the  analytic 
process  involved  in  developing  this  chart 
is  included  below. 

In  Game  1,  both  sides  are  assumed  to 
have  common  knowledge  of  the  values 
associated  with  their  strategies  —  i.e.. 


both  sides  know  what  the  random  num¬ 
bers  are  that  appear  in  all  rows  and 
columns  of  the  3x3,  5x5,  and  10x10 
matrices  created  by  their  opposing  strate¬ 
gies.  In  other  words,  both  sides  have  the 
same  information  and  neither  benefits 
from  information  superiority.  Since  the 
underlying  payoffs  are  random  and  dis¬ 
tributed  uniformly  between  0  and  100,  the 
expected  payoff  is  50.  Any  other  result  of 
the  1000  trials  would  indicate  that  our 
computer  program  was  incorrect. 

In  Game  2,  Side  1  (the  US)  knows  the 
values  associated  with  the  strategies  of 
both  sides  —  i.e.,  it  has  information  supe¬ 
riority  —  but  Side  2  does  not;  it  has  bad 
information.  This  situation  may  be  simi¬ 
lar  to  the  one  that  US  Army  expects  to 
face  during  the  first  decades  of  the  21st 
century,  when  the  US  will  be  dealing  with 
potential  enemies  who  seem  likely  to  lack 
the  Army’s  highly  digitized  and  intemet- 


Dr.  Richard 
Darilek 

RAND 


Information  Superiority  and  Game  Theory: 
Probabilities  of  Victory 
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Game  1 :  Both  sides  have  common  knowledge  of  values  associated  with  their  strategies 


Game  2:  Side  1  knows  values  associated  with  strategies  of  both  sides.  Side  2  does  not. 


Game  3:  Both  sides  have  common  knowledge  of  values  associated  with  strategies. 
Side  1  knows  Side  2’s  choice  of  strategy. 


Game  4:  Side  1  knows  values  associated  with  strategies  of  both  sides.  Side  2  does  not. 
Side  1  knows  Side  2’s  choice  of  strategy. 

Arroyo  Center  RAND 


PHALANX 


6 


December  1998 


ted  units  (i.e.,  the  Army  XXI  forces  cur¬ 
rently  planned  to  be  in  the  field  around 
2010). 

In  Game  3,  both  sides  have  common 
knowledge  of  the  values  associated  with 
their  strategies,  but  Side  1  (the  US)  knows 
Side  2’s  choice  of  strategy.  In  other 
words,  Side  1  possesses  perfect  intelli¬ 
gence  and,  as  a  result,  another  kind  or 
higher  level  of  information  superiority, 
even  though  Side  2’s  basic  information  in 
this  case  (as  opposed  to  Game  2)  is  not 
bad.  This  situation  reminds  us  of  a  capa¬ 
bility  proposed  for  the  Army  After  Next 
(AAN),  which  the  Army  hopes  the  RMA 
will  have  enabled  it  to  develop  by  about 
2020  or  2025;  the  AAN  aims  to  have 
such  perfect  intelligence. 

In  Game  4,  Side  1  (the  US)  knows  the 
values  associated  with  the  strategies  of 
both  sides,  Side  2  does  not  know,  and 
Side  1  knows  Side  2’s  choice  of  strategy. 
In  this  instance,  Side  1  has  both  perfect 
information  and  perfect  intelligence;  it 
may  even  have  established  this  position 
by  actively  ensuring  (through  offensive 
information  operations,  for  example)  that 
Side  2  has  bad  information.  Thus,  Side  1 
enjoys  not  only  information  superiority 
but  also  what  might  be  called  information 
dominance.  This  recalls  one  of  the  key 
objectives  not  only  of  the  AAN  but  also 
of  the  Army  as  a  whole,  which  pro¬ 
claimed  in  Army  Vision  2010  that  “Gain¬ 
ing  Information  Dominance  ....  is  funda¬ 
mental  to  ...  each  of  the  operational 
concepts  in  Joint  Vision  2010^^ 

Of  particular  interest  is  Game  4.  With 
10  strategies  on  both  sides,  Side  I’s  prob¬ 
ability  of  victory  rises  to  91.5,  an  out¬ 
come  due  entirely  to  Side  I’s  absolute 
information  superiority  and  dominance  in 
this  case.  What  the  outcome  suggests  is 
that  control  of  information  is  the  single 
most  decisive  key  to  victory  for  the  US 
military  in  the  future.  Correspondingly, 
lack  of  information  superiority  in  the 
future  could  prove  to  be  devastating  to  the 
side  that  finds  itself  in  the  inferior  posi¬ 
tion. 

Also  implicit  in  these  games  is  the 
notion  that  information  superiority  and 
dominance  result  from  dynamic  interac¬ 
tions  between  the  two  sides.  These  can 
change  over  time  —  e.g.,  during  the 
course  of  a  conflict.  Hence,  we  should 
guard  against  thinking  of  information 
superiority  or  dominance  as  static  condi¬ 
tions  that,  once  obtained,  endure  for  one 


side  or  the  other  indefinitely.  Prudence 
suggests  viewing  them  as  military  objec¬ 
tives  to  be  achieved  or  restored,  perhaps 
only  temporarily. 

These  games,  of  course,  represent  rela¬ 
tively  simple,  abstract  calculations.  In 
heu  of  harder  data  on  21st  century  forces, 
which  it  is  still  too  soon  to  expect,  such 
games  simply  help  demonstrate  the  theo¬ 
retical  or  potential  contribution  of  infor¬ 
mation  superiority  and  information  domi¬ 
nance  to  victory.  Although  they  are 
instructive,  these  abstract  calculations  and 
theoretical  models  cannot  yet  serve  as  a 
reliable  basis  for  policy  or  programming 
choices.  Prudence  and  common  sense 
suggest  that  a  series  of  carefully  struc¬ 
tured  and  controlled  field  experiments 
will  be  needed  to  test  the  vahdity  of  any 
theoretical  model’s  results. 

ANALYSIS 

This  process  shows,  for  a  very  simple 
and  easy-to-understand  situation,  how  the 
outcome  of  a  zero-sum  game  changes  as 
the  information  of  the  two  sides  changes. 

In  addition  to  four  different  basic 
assumptions  about  the  information  avail¬ 
able  to  both  sides  we  consider  three  cases 
of  dimensionality  with  respect  to  the 
number  of  strategies  or  choices  available 
to  both  sides.  We  allow  each  side  three, 
five,  or  ten  choices.  The  effects  of  infor¬ 
mation  differ  depending  on  this  character¬ 
istic  of  the  game.  (This  feature  of  the 
game  has  some  intuitive  relationship  with 
warfare  because  a  commander,  or  com¬ 
batant,  can  only  logically  process  a  cer¬ 
tain  amount  of  information  in  a  stressful 
situation,  which  may  lie  in  such  low¬ 
dimensional  regions  as  these.) 

We  also  perform  a  sensitivity  analysis 
of  the  effect  of  one  side’s  using  a  non- 
optimal  decision  rule  rather  than  an  opti¬ 
mal  strategy.  This  provides  insight  into 
the  differences  between  decision  process¬ 
es  based  on  decision  mles  versus  decision 
processes  based  on  game-theoretic  solu¬ 
tions. 

Description  of  Two-Sided 
Game  and  its  Solution 

Side  1  has  choices  i  =  l,...,m  and  Side 
2  has  choices  j  =  l,...,n.  For  each  choice 
there  is  a  payoff  ajj.  Side  1  receives  a^j 
and  Side  2  loses  ajj.  Side  1  wishes  to 
maximize  the  payoff  and  Side  2  wishes  to 
minimize  the  payoff. 


The  choice  of  the  optimal  action  i*  of  Side 
1  is  made  by  computing,  for  each  choice 
of  Side  2,  the  minimum  that  Side  1  can 
gain,  or: 

aj™^  =  minimum  ay, 

j 

and  then  choosing  the  i*  which  maximizes 
the  minimum  that  Side  1  can  gain,  or: 

i*  =  i  corresponding  with 
maximum  a^™^. 

This  is  the  maxmin  strategy  of  Side  1. 

Similarly,  the  choice  of  the  optimal 
action  j*  of  Side  2  is  made  by  computing, 
for  each  choice  of  Side  1,  the  maximum 
which  Side  2  can  lose,  or: 

ajmax  _  maximum  ay, 
i 

and  then  choosing  the  j*  which  minimizes 
the  maximum  that  side  2  can  lose,  or: 

j*  =  j  corresponding  with 
minimum 

j 

This  is  the  minmax  strategy  of  Side  2. 

For  the  Side  1  choice  i*  and  Side  2  choice 
j*  the  payoff  of  the  game  is: 

Payoff  of  Game  =  aj*j*. 

This  payoff  is  guaranteed  to  be  equal  to 
or  greater  than  the  maxmin  payoff  corre¬ 
sponding  to  i*  and  equal  to  or  less  than  the 
minmax  payoff  corresponding  to  j*.  Side 
1  receives  at  least  his  maxmin  payoff  and 
Side  2  loses  at  most  his  minmax  payoff. 

The  specific  implementation  of  the 
game  explored  in  this  paper  is  to  generate 
using  random  numbers  payoffs  from  0  to 
100.  The  Side  1  maxmin  strategy  and 
Side  2  minmax  strategy  are  found  and  the 
payoff  associated  with  this  strategy  pair  is 
saved.  The  process  is  repeated  KXX)  times 
and  the  average  payoff  computed. 

Three  cases  are  studied,  as  follows: 

3X3  game  with  9  payoffs 
5X5  game  with  25  payoffs 
10X10  game  with  100  payoffs. 

(See  GAMES,  p.  33) 
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Introduction 


This  note  is  our  rejoinder  to  Gary 
Coe’s  comments  in  “ADS  for 
Analysis  —  A  Different  View”! 
on  our  “ADS  for  Analysis:  The  Reality 
and  the  Hype!”^  article.  In  particular,  we 
want  to  clarify  a  couple  of  important 
points  that  were  misinterpreted  by  Mr. 
Coe.  We,  in  fact,  concur  with  many  of 
his  positions.  There  are,  however,  some 
important  differences.  These  arise  from 
considerations  on  how  models  and  simu¬ 
lations,  including  Advanced  Distributed 
Simulation  (ADS),  should  be  used  within 
an  analysis  to  best  inform  the  decision 
making  process.  Furthermore,  we 
strongly  disagree  with  Mr.  Coe  on  the 
magnitude  of  the  analytic  challenges 
associated  with  reliable  ADS-based 
experiments  and  their  potential  effects  on 
an  analysis.  As  a  consequence,  we 
emphasize  the  need  to  carefully  heed 
“historical”  design  of  experiment  princi¬ 
ples  when  conducting  experiments  using 
ADS. 

We  believe  that  ADS  technologies 
will  be  an  increasingly  important  analyt¬ 
ic  tool  and  our  intent  is  to  help  ensure 


Much  Agreement,  Important  Differences 


that  their  analytic  potential  is  realized. 
We  want  to  avoid  excessive  hype  of 
“ADS  for  analysis”  followed  by  dashed 
expectations  —  and  reduced  investment. 

Clarifying  a  Few  Points 

Since  ADS  is  becoming  an  important 
analysis  tool  it  is  valuable  to  understand 
where  analysts  agree  and  disagree. 
Towards  that  end  we  begin  by  reviewing 
some  of  Coe’s  mischaracterizations  of 
our  views  on  important  aspects  of  analy¬ 
sis  and  ADS.  These  include  subtle 
wording  differences  that  change  our 
intended  meaning,  and  inaccurately 
attributing  positions  to  us  on  topics  we 
did  not  address  in  our  original  PHA¬ 
LANX  article.  A  more  extensive  treat¬ 
ment  of  “ADS  for  Analysis”  can  be 
found  in  “A  Guide  to  Analysis  Using 
Advanced  Distributed  Simulation''^  and 
'''Understanding  the  Air  Force's  Capa¬ 
bility  to  Effectively  Apply  Advanced  Dis¬ 
tributed  Simulation  for  Analysis"^.  Now, 
let’s  clarify  some  important  “ADS  for 
Analysis”  issues. 

Coe:  Although  the  RAND  idea  of  using 
human-in-the-loop  experiments  to  cali¬ 
brate  constructive  simulations  may  have 
merit,  it  is  believed  that  this  view  is  too 
narrow  as  a  scheme  for  improved  use  of 
ADS  in  analysis. " 

We  agree,  which  is  part  of  the  reason 
we  were  careful  never  to  write  “cali¬ 
brate”  in  "ADS  for  Analysis:  The  Reality 
and  the  Hype!".  We  write  that  one  of  the 
important  analytic  uses  is  to  use  “ADS 
runs  [to]  inform  human  performance  fac¬ 
tors  and  other  factors  in  constructive 
models.”  The  other  factors  (perhaps  how 
battles  evolve  or  operational  concepts 
associated  with  new  systems  or  scenar¬ 
ios)  may  be  more  important;  this  will  be 
highly  dependent  on  specific  study 
objectives.  We  also  believe  that  the 
ADS  runs  can  provide  valuable  infonna- 
tion  in  and  of  themselves.  We  state  that 
the  few  ADS  runs  are  “the  use  most  visi¬ 
ble  to  the  consumers  of  the  analysis  and 
correspond  to  high-value  ADS  runs  for 
scenarios  of  interest.”  There  are  other 
good  uses  for  ADS  as  well,  many  yet  to 
be  discovered. 


Coe:  "The  RAND  article  assumes  that 
constructive  models  and  ADS  have 
uniquely  different  attributes  that  need  to 
be  normalized  in  some  way. " 

We  make  no  such  assumption.  Con¬ 
structive  models  are  an  important  ele¬ 
ment  within  ADS.  The  ADS  vision,  as 
articulated  in  "ADS  for  Analysis:  The 
Reality  and  the  Hype!",  is  that  one  can 
“realistically  simulate.. .with  joint,  live, 
virtual  and  constructive  elements  partici¬ 
pating.”  In  many  ADS  applications  con¬ 
structive  models  simulate  the  majority  of 
entities.  Our  point  is  that  models  “with 
different  algorithms,  approaches,  purpos¬ 
es  and  different  levels  of  resolution”  will 
almost  surely,  when  interacting  on  the 
synthetic  battlefield,  generate  biased  out¬ 
comes.  That  is,  some  models,  systems, 
players  and  entities  will  have  unintended 
systematic  advantages  over  other  simula¬ 
tion  elements.  This  is  sometimes  called 
an  “unfair  fight.”  These  inevitable  bias¬ 
es,  if  not  understood  and  accounted  for, 
may  lead  to  erroneous  insights.  While 
potential  sources  of  bias  are  a  concern 
with  many  simulations,  the  nature  of 
ADS  greatly  exacerbates  the  problem. 

Coe:  "Rather  than  think  about  ADS  in 
terms  of  special  properties  such  as  dis¬ 
tributed  simulation,  it  is  more  important 
to  understand  how  each  of  these  tech¬ 
nologies.. .may  contribute  to  analysis. " 

We  emphasize  distributed  simulation 
because  it  is  the  primary  “new”  develop¬ 
ment.  One  of  then  -  Lt  Col  Bob 
Sheldon’s  working  group’s  primary 
findings  at  the  1996  MORS  ADS  for 
Analysis  Workshop  was  that  “ADS  is 
not  a  new  methodology  for  analysis  — 
all  the  components,  including  human-in- 
the-loop,  have  been  available  to  analysts 
for  years.  Rather,  it  is  glue  that  allows 
analysts  to  tie  together  formerly  stand¬ 
alone  analytic  tools  into  a  more  powerful 
mechanism  for  modeling  complex  prob¬ 
lems.”  Simulation  elements  like  “Syn¬ 
thetic  Forces”  and  “Synthetic  Environ¬ 
ments”  have  existed  in  ever  more 
detailed  versions  for  decades. 

Coe:  "The  RAND  paper  cites  the  value 
of  visualization  in  communicating  analy- 
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sis  results  to  the  decision  maker.  Why  is 
it  not  equally  important  to  analyze 
processes  visually...  ?” 

We  don’t  disagree,  but  simply  had 
insufficient  space  in  our  earlier  PHA¬ 
LANX  article  to  enumerate  all  the  values 
of  all  ADS  attributes.  The  visualization 
that  is  inherent  (though  not  unique)  in 
ADS  is  one  of  its  primary  benefits  and 
should  be  exploited  in  many  ways.  In  “A 
Guide  to  Analysis  Using  Advanced  Dis¬ 
tributed  Simulation”^  we  write  “Visual¬ 
ization  tools  [are  part  of  the]  summary 
and  review  process,  although  their  use,  of 
course,  transcends  just  this  application. 
The  authors  have  been  involved  in  con¬ 
structive  simulation  study  efforts... where 
for  all  intents  and  purposes  the  only  quick 
review  tool  we  really  needed  was  a  visual 
playback  of  the  scenarios  that  had  just 
been  run.” 

Coe:  *'It  is  believed  that  beyond  tradition¬ 
al  analysis,  ADS  benefits  a  larger  domain 
than  only  the  human  dimension  —  specif¬ 
ically  the  information  domain.  ” 

Our  emphasis  on  ADS’s  facilitating 
the  incorporation  of  real  warfighters  in 
simulations  should  not  be  construed  to 
imply  that  only  studies  “on  the  human 
dimension”  benefit  from  ADS.  Rather, 
any  domain  where  human  performance  is 
critical  may  potentially  benefit.  This  is  a 
vast  area;  for  example,  we  noted  in  our 
earlier  PHALANX  paper  that  it  has  been 
stated  by  Marshall  and  Garrett^  that 
“ADS  should  be  particularly  useful  in 
modeling  C4ISR.” 

It  turns  out  that  we  have  more  in 
agreement  with  Mr.  Coe  than  his  article 
suggests.  Most  of  the  examples  he  cites 
as  ADS  advantages  are  simply  specific 
examples  (e.g.,  STOW  97)  of  some  of  the 
more  general  analytic  ADS  advantages 
we  listed.  However,  the  points  of  dis¬ 
agreement  are  more  interesting  and  we 
will  now  address  a  few  important  ones. 

On  Analysis,  New  Paradigsm, 
Experimental  Design  Principles, 
and  Decision  Making 

Effective  use  of  simulation  experi¬ 
ments  in  an  analysis  depends  on  the 
nature  and  context  of  the  analysis.  For 
this  note  we  take  the  perspective  that 
analysis  is  the  process  of  learning  about 


complex  systems  and  their  interactions  to 
improve  decision  making.  Analyses  are 
usually  commissioned  because  someone 
needs  to  make  a  decision  among  some 
group  of  choices  —  and  subsequently  to 
help  justify  the  option  selected.  Here, 
“decision”  is  used  very  broadly;  ranging 
from  specific  materiel  acquisition  deci¬ 
sions  to  determining  what  general  opera¬ 
tional  concepts  should  be  explored  fur¬ 
ther.  A  successful  analysis  is  one  that 
produces  a  compelling  argument  for 
making  the  best  choice  from  the  avail¬ 
able  options. 


Mlitaiy  analysis,  as  well  as 
analysis  in  other  public  policy  areas, 


is  not  an  exact  science 
because  of  the  vast  number  of 
unknowns  exphcitly  or  imphcitly  cast 
as  assumptions  and  assertions. 


The  critical  question  is:  What  role 
can/should  simulation  experiments  play 
in  synthesizing  an  argument  for  or 
against  a  potential  decision?  Military 
analysis,  as  well  as  analysis  in  other  pub¬ 
lic  policy  areas,  is  not  an  exact  science 
because  of  the  vast  number  of  unknowns 
explicitly  or  implicitly  cast  as  assump¬ 
tions  and  assertions.  Among  other  uses, 
simulation  experiments  can  provide 
information  on  these  unknowns.  Simula¬ 
tion  experiments  are  usually  the  most 
informative  if  they  are  designed  to  assess 
specific  load-bearing  hypotheses  gleaned 
from  unknowns  in  a  tentative  argument 
constructed  to  (potentially)  make  a  deci¬ 
sion. 

The  history  of  human  discovery  is  rife 
with  examples  of  poor  or  erroneous  con¬ 
clusions  drawn  from  improper  use  of 
experimental  results.  Our  primary  areas 
of  disagreement  with  Mr.  Coe  centers  on 
how  one  should  reason  given  the  pre¬ 
sumed  “realism”  of  combat  simulations 
and  the  resultant  importance  of  tradition¬ 


al  design  of  experiment  principles  as  a 
tool  to  efficiently  learn  from  simulation 
experiments.  ADS  is  certainly  a  new 
capability  for  performing  mixed  live  and 
computational  experiments.  Although 
new  methods  may  be  developed,  we 
believe  that  full  advantage  must  be  taken 
of  experimental  design  techniques  to 
minimize  errors  in  the  decision  making 
process  and  to  strengthen  the  arguments 
that  utilize  ADS.  We  worry  that  if  ana¬ 
lysts  put  too  much  faith  in  “vicarious” 
simulation  experiments  and  do  not  sup¬ 
port  the  analysis  with  “traditional  meth¬ 
ods”  the  insights  gleaned  by  visualizing 
the  processes  within  a  few  high-resolu¬ 
tion  STOW  simulations  may  be  persua¬ 
sive  illusions  that  foster  poor  decision 
making. 

Mr.  Coe  notes  that  analysts  help  deci¬ 
sion  makers  by  providing  “insight”  and 
“information,”  rather  than  the  “right 
answer.”  In  general,  we  agree.  We  may 
disagree,  however,  on  how  this  is  done. 
In  his  article,  while  describing  STOW 
97,  Mr.  Coe  uses  the  words  “realistic”  or 
“good  fidelity”  more  than  half  a  dozen 
times.  While  many  of  the  subcompo¬ 
nents  may  be  good  models  or  even  the 
actual  equipment,  due  to  a  dearth  of 
empirical  data,  a  belief  that  any  simula¬ 
tion  of  potential  future  campaigns  is 
“realistic”  relies  more  on  faith  than 
demonstrable  evidence.^  There  are 
countless  potentially  causal  unknowns 
and  uncertainties  (e.g.,  a  threat’s  tactics 
and  its  will  to  fight,  systems’  effective¬ 
ness  and  reliabilities,  environmental  fac¬ 
tors,  other  stochastic  realizations,  etc.),  in 
addition  to  inevitable  model  biases,  that 
necessarily  were  not  addressed  in  STOW 
97. 

STOW  97-type  simulations  facilitate 
looking  at  a  few  high-resolution  (not 
necessarily  high-fidelity)  cases  — 
though  subevents  and  processes  may 
occur  many  times.  But,  most  combat 
simulations,  such  as  STOW  97,  are  not 
vahdated  in  the  sense  that  their  outcomes 
can  credibly  be  considered  predictions  of 
potential  real-world  outcomes  with 
known  accuracy.  Thus,  bad  decisions 
can  result  from  arguments  that  at  their 
core  assume  the  validity  of  these  unvali¬ 
dated  simulations.  Dewar  et  al.  gives 
several  approaches  to  credible  reasoning 
with  nonpredictive  simulations.^  Most 

(See  p.  10) 
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ADS 

(continued from  p.  9) 

of  these  approaches  require  running  a 
large  number  of  cases  (i.e.,  varying  lots  of 
uncertain  things)  with  the  veracity  of  any 
individual  case  being  of  lesser  impor¬ 
tance. 

Concerning  new  analytic  paradigms, 
Mr.  Coe  writes  that  there  might  be  “new 
approaches  to  data  analysis  such  as  in 
greater  dependence  on  visualization  vice 
statistical  analysis.”  We  are  big  fans  of 
visualization.  However,  we  believe  that 
more  rather  than  less  statistical  analysis  is 
needed  to  help  us  decipher  whether 
apparent  patterns  are  real  or  can  be 
explained  by  the  random  variation  inher¬ 
ent  in  battle  (or  data  mining).  As  Efron 
and  Tibshirani  write,  “Left  to  our  own 
devices  we  are  not  very  good  at  picking 
out  patterns  from  a  sea  of  noisy  data.^  To 
put  it  another  way  we  are  all  too  good  at 
picking  out  non-existent  patterns  that  hap¬ 
pen  to  suit  our  purposes.” 

Finally,  as  to  needing  “to  develop  new 
(analytic)  processes.”  We  agree,  and  pre¬ 
sented  one  such  possibility  in  ""ADS  for 
Analysis:  The  Reality  and  the  HypeT', 
however,  we  argue  strongly  that  the  new 
processes  must  be  mindful  of  why  some 
of  the  “traditional”  approaches  were 
developed.  Over  time  the  scientific  com¬ 
munity  has  adopted  several  experimental 
design  principles.  Some  important  ones 
are  called  randomization,  replication, 
blinding,  blocking,  control,  sensitivity 
analysis  and  peer  review.  These  were 
learned  by  carefully  examining  why  some 
important  experiments  produced  erro¬ 
neous  conclusions.  The  cause  of  error  in 
an  experimental  design  can  be  very  sub¬ 
tle,  such  as  the  Hawthorne  Effect;  see 
Roethlisberger.^ 

Freedman  et  al.  and  Schulz  et  al. 
have  excellent  discussions  on  the  conse¬ 
quences  of  violating  design  of  experiment 
principles. They  review  several 
examples  that  relate  the  conclusions  of  a 
large  number  of  similar  studies  to  how 
well  they  were  designed.  They  reveal  a 
strong  inverse  relationship  between  the 
studies’  conclusions  on  the  efficacy  of  a 
variety  of  (medical)  procedures  and  the 
quality  of  the  designs  (as  measured  by 
some  of  the  above  principles).  Schulz  et 
al.  writes:  ‘This  study  provides  empirical 
evidence  that  inadequate  methodological 
approaches. .  .are  associated  with  bias.”  It 


can  be  hazardous  to  assume  that  the 
defense  community  is  somehow  immune 
to  these  experimental  challenges.  An 
illuminating  discussion  on  how  erro¬ 
neous  insights  were  obtained  on  anti¬ 
tank  tactics  and  system  effectiveness 
(and  the  subtle  biases  that  caused  the 
errors)  in  the  1941  Louisiana  Maneuvers 
is  contained  in  Gabel.  It  is  simply  an 
established  fact  that  intelligent  well- 
intentioned  people  with  area  expertise 
can  make  had  decisions  ( in  terms  of  lost 
money  or  lives)  when  their  experiments 
violate  the  above  principles.  Thus,  the 
need  to  support  new  analytical  capabili¬ 
ties  (such  as  STOW  experiments)  with 
traditional  experimental  principles  and 
methods. 


We  applaud  Mr.  Coe  for  his 
vigorous  support  of  ADS  technologies, 
and  for  articulating  what  was  achieved 
in  STOW  97.  We  see  many 
of  the  potential  analytic  benefits 
he  does  and  hope  they 
arereahzed. 


Final  Thoughts 

We  applaud  Mr.  Coe  for  his  vigorous 
support  of  ADS  technologies,  and  for 
articulating  what  was  achieved  in  STOW 
97.  We  see  many  of  the  potential  analyt¬ 
ic  benefits  he  does  and  hope  they  are 
realized.  Indeed,  we  believe  that  ADS 
technologies  are  critical  in  our  efforts  to 
understand  the  evolving  vision  of  future 
warfare  as  articulated  in  Joint  Vision 
2010.^^  Towards  that  objective  we  trust 
that  this  rejoinder  helps  illuminate  those 
areas  in  which  we  disagree  about  how  to 
best  achieve  ADS’s  analytic  potential. 
As  with  any  radically  new  capability 
advances  will  not  be  made  without  a 
healthy  debate.  We  believe  that  the 
debate  is  most  productive  if  we  carefully 
analyze  the  foundations  of  the  differ¬ 
ences. 
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Introduction 

In  the  book  Alice  in  Wonderland,  by 
Lewis  Carroll,  the  following  conver¬ 
sation  takes  place  between  Alice  and 
the  Cheshire  Cat: 


“Would  you  tell  me,  please,  which  way  I 
ought  to  go  from  here?” 

‘That  depends  a  good  deal  on  where  you 
want  to  get  to,”  said  the  Cat. 

“I  don’t  much  care  where — ,”  said  Alice. 
“Then  it  doesn’t  much  matter  which  way 
you  go,  “  said  the  Cat. 

“ — so  long  as  I  get  somewhere,”  Ahce 
added  as  an  explanation. 

“Oh,  you’re  sure  to  do  that,  if  you  only 
walk  long  enough”  concluded  the  Cat. 

In  the  early  1990’s  the  Modeling  and 
Simulation  (M&S)  community  was  travel¬ 
ing  in  a  direction  similar  to  that  of  Alice: 
we  knew  we  wanted  to  go  somewhere  but 
we  didn’t  know  exactly  where.  As 
Department  of  Defense  (DoD)  resources 
shrank,  and  the  use  of  M&S  as  an  analyti¬ 
cal  tool  expanded,  it  became  obvious  that 
we  could  no  longer  continue  to  “walk  long 
enough.”  To  meet  this  challenge,  person¬ 
nel  in  the  Army  applied  a  systems 
approach  to  the  problem.  We  examined 
the  current  state  of  the  Army’s  M&S  envi¬ 
ronment,  or  more  simply  put,  where  we 
were.  Next  we  articulated  the  desired 
state,  or  where  we  wanted  to  be,  in  the 
form  of  an  objective  M&S  environment. 
We  then  set  about  establishing  a  course  of 
action  to  bridge  the  gap  between  the  cur¬ 
rent  state  and  the  desired  state.  This 
implementation  embraces  the  development 
of  M&S  standards  through  a  bottom-up 
approach  with  decentralized  authority. 
This  article  will  describe  the  Army’s 


process  to  develop  and  promulgate  techni¬ 
cal  standards  and  how  this  endeavor  sup¬ 
ports  Simulation  Based  Acquisition  (SBA) 
and  Verification  and  Validation  (V&V) 
and  Accreditation. 


In  the  early  1990’s  the 


;  Modeling  and  Simulation  (M&S) 
community  was  traveling  in  a 
direction  similar  to  that  of  Mce; 
we  knew  we  wanted  to  go  somewhere 
but  we  didn’t  know  exactly  where. 

What  Is  An  Army  M&S  Standard? 

Webster’s  II  New  Riverside  University 
Dictionary  defines  standards  as  “a  rule, 
principle,  or  measurement  established  by 
authority,  custom,  or  general  consent  as  a 
representation  or  example.”  Following 
this  definition,  the  Army’s  applies  the  term 
M&S  standard  in  the  broadest  context  to 
include  procedures,  practices,  processes, 
techniques,  algorithms  and  heuristics. 
Standards  for  M&S  cover  a  variety  of  top¬ 


ics  and  the  type  and  source  of  relevant 
standards  will  vary  with  each  of  the  19 
standards  categories.  The  standards  cate¬ 
gories  represent  the  M&S  functions  that, 
taken  as  a  whole,  cover  the  technological 
spectrum. 

Simply  put,  the  Army  seeks  to  develop 
standards  to  improve  M&S  interoperability 
and  credibility  while  also  increasing  com¬ 
monality  and  reuse.  Through  the  develop¬ 
ment  of  M&S  standards  the  Army  hopes 
to: 

•  Enable  simulations  to  provide  or 
accept  services  from  one  another 
thus  making  them  more  interopera¬ 
ble; 

•  Improve  the  credibility  or  accep¬ 
tance  of  M&S  representations; 

•  Increase  commonality  in  the  depic¬ 
tion  of  the  synthetic  environment; 
and, 

•  Establish  a  baseline  for  reusing  stan¬ 
dard  algorithms  and  heuristics  in 
future  simulations. 

The  Army’s  Seven  Step  M&S 
Standards  Development  Process 

The  Army’s  development  of  M&S 
standards  is  consensus-based  by  choice. 
M&S  technologies  evolve  at  blinding 
speeds.  Some  technological  niches  turn 
over  in  a  matter  of  months.  Technological, 
(SeeM&S,p.  12) 


The  Army’s  19  M&S  Standards  Categories 


•  Architecture 

•  C4I  Integration 

•  Data 

‘  Functional 
Description  of  the 
Battlespace 


•Object  Management 


•Semi-Automated  Forces 
•Visualization 


•Dynamic  Atmospheric  Environments 
•Terrain 


•Deployment/Redeployment 
•Mobilization/Demobilization 
•Communication  Systems 


•Acquire 

•Attrition 

•Move 

•Logistics 


•Command  Decision  Modeling 


•Cost  Representation 


•  VV&A 


Analysis 


Acquisition 


Training 


Standards  for  Developing 
Simulations  and  Data 


1  I 


Standards  for  Modeling 
the  Environment 


Standards  for  Modeling 
Operations/Phenomenology 

"II  "  "1  I . 


Standards  for  Modeling 
Cognitive  Processes 


Standards  for 
Ensuring  Credibility 


FIGURE  1.  The  Army’s  M&S  Standards  Categories 
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M&S 

(continued  from  p.  11) 

procedural  and  application  advances  take 
place  within  a  myriad  of  organizations  in 
the  Army,  and  throughout  the  government, 
commercial  and  academic  sectors. 
Attempting  to  centralize  the  authority  for 
establishing  standards,  without  devoting 
major  resources  to  the  effort,  would  not 
enable  the  Army  to  remain  a  M&S  com¬ 
munity  leader.  Rather,  it  could  place  the 
Army  permanently  behind  the  rest  of  the 
M&S  community.  No  single  Army  office 
or  organization  is  capable  of  effectively 
investigating  and  making  the  necessary 
decisions  to  evolve  Army  models  and  sim¬ 
ulations  that  keep  pace  with  the  rest  of  the 
industry.  By  keeping  the  process  consen¬ 
sus-based,  those  decisions  are  in  the  hands 
of  the  real  Army  M&S  experts. 

Build  Teams.  Subject  matter  experts 
from  various  organizations  throughout  the 
Army  are  appointed  to  serve  as  Standards 
Category  Coordinators  (SCCs).  They  are 
not  executive  agents.  They  serve  as  the 
leadership  for  developing  M&S  standards 
within  their  individual  category.  They  are 
empowered  to  develop  their  teams  by 
drawing  on  the  mix  of  talents  and  expertise 
needed  in  their  specific  area.  Team  com¬ 
position  is  interdisciplinary  and  knows  no 
organizational  boundaries.  Membership  is 
based  on  inclusion  rather  than  exclusion. 
Each  team  has  an  electronic  mail  reflector, 
which  permits  a  wider  community  to  par¬ 
ticipate  in  the  development  of  future  stan¬ 
dards  while  minimizing  travel. 

Define  Requirements.  The  second  step 
is  to  define  requirements.  To  assist  in  this 
and  the  next  three  steps  the  Army  has 
developed  the  Standards  Nomination  and 
Approval  Process  (SNAP)  at 
http://www.msrr.army.mil/snap.  At  the 
heart  of  SNAP  is  the  Standards  Require¬ 
ment  Document  (SRD).  The  SRD,  an  on¬ 
line  form,  is  the  first  step  in  developing  a 
new  Army  M&S  standard;  refining  an 
existing  standard;  or  nominating  an  accept¬ 
ed  M&S  procedure,  practice,  process,  tech¬ 
nique,  algorithm  or  heuristic  to  become  a 
standard.  With  only  limited  resources  to 
devote  to  the  development  of  standards,  it 
is  essential  to  keep  the  work  of  the  team 
focused  on  the  most  important  issues. 
Once  an  SRD  is  received,  the  Army  Model 
and  Simulation  Office  (AMSO)  coordi¬ 


nates  validation  of  the  proposed  require¬ 
ment  with  the  appropriate  SCC  and  other 
key  players  to  ensure  that  the  proposal 
supports  a  community  need.  After  valida¬ 
tion  is  completed,  the  initiator  is  provided 
feedback.  A  proposed  standard  could  be 
an  excellent  idea  but  may  fail  to  fit  a  “mar¬ 
ket  niche.”  If  the  proposal  is  approved,  it 
then  moves  on  to  the  next  step  in  the 
process. 


of  the  proposed  requirement 
with  the  appropriate  SCC 
and  other  key  players  to  ensure 
that  the  proposal  supports 
a  community  need. 

Develop  Standards.  This  step  is  the 
crux  of  the  Army’s  M&S  Standards 
Development  Process.  The  wider  the 
involvement  of  experts  across  the  M&S 
community,  the  more  likely  each  team  will 
capture,  adapt,  or  develop  those  proce¬ 
dures,  processes,  techniques,  algorithms 
and  heuristics  —  as  well  as  “best  and  cur¬ 
rent  practices”  —  that  warrant  becoming 
an  Army  M&S  standards. 

Achieve  Consensus.  After  the  standard 
has  been  developed,  the  next  step  is  to 
achieve  consensus.  SNAP  is  the  primary 
vehicle  used  by  the  teams  to  “hammer- 
out”  issues  and  achieve  consensus  on  a 
draft  standard.  SNAP  monitors  all  elec¬ 
tronic  mail  traffic  on  the  reflectors  and  as 
part  of  its  database  maintains  a  copy  of 
every  reflected  message  sent.  This  useful 
feature  allows  one  to  review  the  on  going 
debate  on  one  or  more  draft  standards, 
enter  the  debate  at  any  time  during  the 
process,  and  eliminates  the  need  for  team 
members  to  personally  maintain  a  copy  of 
every  electronic  mail  message.  Each  draft 
standard  may  go  through  several  iterations 
before  being  embraced  by  the  team. 


Because  the  process  is  continuous  and  iter¬ 
ative,  the  community  more  readily  adopts 
standards  they  feel  can  be  modified  and 
improved  over  time. 

Obtain  Approval.  Once  consensus  has 
been  achieved,  a  panel  of  Senior  M&S 
subject  matter  experts  then  reviews  the 
draft  standard.  SNAP  automatically  sends 
an  electronic  mail  message  to  the  appropri¬ 
ate  reviewers.  The  message  contains  a 
“hot-link”  to  their  individual  voting  page 
along  with  information  on  the  draft  stan¬ 
dard.  Senior  reviewers  may  either  vote 
“Yes”  or  “No.”  If  they  vote  “No”  a  com¬ 
ment  field  must  be  completed  or  the  vote 
will  not  be  accepted.  Every  effort  will  be 
made  to  resolve  the  senior  reviewer’s  con¬ 
cerns  prior  to  the  closure  of  voting.  After 
voting  has  concluded  the  proposed  stan¬ 
dard  and  all  comments  are  either  forward¬ 
ed  to  the  Deputy  Under  Secretary  of  the 
Army  (Operations  Research)  (DUSA(OR)) 
for  approval  or  returned  to  the  standards 
category  team  for  additional  work. 

Promulgate  Standards.  Every 
approved  Army  M&S  standard  will  be  reg¬ 
istered  in  the  Army  Standards  Repository 
System  (ASTARS)  at  http://www.msrr. 
army.mil/astars.  For  each  entry,  you  will 
find  information  about  the  standard  and  a 
point-of-contact.  To  the  maximum  extent 
practical  standards  will  be  made  available 
electronically.  Standards  in  ASTARS  can 
be  password-protected  when  access  needs 
to  be  limited.  However,  classified  stan¬ 
dards  will  not  be  stored  in  the  current  ver¬ 
sion  of  ASTARS,  Those  standards  not 
available  for  public  release  will  follow  the 
release  procedures  for  M&S  described  in 
Army  Regulation  5-11.  Each  SCC  has  a 
Home  Page  to  provide  specific  information 
pertaining  to  their  category.  Information 
on  all  the  aspects  of  M&S  Standards  can 
also  be  accessed  from  both  the  AMSO 
Home  Page  (http://www.amso.army.mil) 
and  the  Army  Node  of  the  Modeling  and 
Simulation  Resource  Repository  (MSRR) 
(http://www.msrr.army.mil). 

Educate.  Educating  and  assisting  mod¬ 
elers  and  users  is  accomplished  concuirent- 
ly  with  the  other  steps.  Once  a  standard  has 
been  approved,  the  team  begins  educating 
the  M&S  community  on  the  availability, 
applicability  and  use  of  the  standard.  The 
more  active  the  standards  category  team, 
the  more  educated  the  community. 
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How  Do  Standards  Expand  M&S 
as  an  Analytical  Tool? 

The  vision  for  SBA  is  “to  have  an 
acquisition  process  enabled  by  robust,  col¬ 
laborative  use  of  simulation  technology 
that  is  integrated  across  acquisition  phases 
and  programs  ”  The  employment  of  M&S 
standards  is  a  key  component  in  achieving 
this  vision.  The  Army’s  Standard  Devel¬ 
opment  Process  provides  a  systematic 
approach  to  the  development,  use  and 
reuse  of  the  components,  as  well  as,  the 
complete  M&S  software.  This  provides 
the  opportunity  to  substantially  reduce 
time,  resources  and  risk  associated  with  the 
acquisition  process.  By  using  approved 
standards,  M&S  developers  have  a  solid 
foundation  upon  which  to  build.  The 
developers  can  begin  their  design  efforts 
by  querying  ASTARS  to  see  if  a  standard 
presently  exists  to  meet  their  needs.  If  not, 
an  SRD  can  be  submitted  to  either  modify 
an  existing  standard  or  create  a  new  one  for 
this  particular  use.  In  addition,  the  Army’s 
M&S  Standards  Development  Process  pro¬ 
vides  a  forum  for  M&S  members  of  the 
analytical,  acquisition  and  training  commu¬ 
nities  to  leverage  one  another’s  work  and 
the  potential  to  promote  Integrated  Product 
and  Process  Development  throughout  a 
systems’  product  lifecycle. 

The  use  of  approved  M&S  standards 
also  benefits  V&V  as  well  as  Accredita¬ 
tion.  Via  the  Army’s  M&S  Standards 
Development  Process,  verification  is  expe¬ 
dited  by  the  fact  that  the  components  of  the 
M&S  have  been  previously  examined  to 
ensure  conformance  to  sound  software¬ 
engineering  techniques.  Validation  is 
enhanced  because  the  standards  have 
already  been  reviewed  by  subject  matter 
experts  and  senior  analysts  to  ensure  the 
standard  in  question  is  a  valid  representa¬ 
tion  of  its  real  world  counterparts.  All 
approved  standards  will  be  documented 
thus  providing  both  V&V  and  Accredita¬ 
tion  agents  information  on  the  utility  and 
limitations  of  a  standard.  In  short,  by  using 
approved  standards,  V&V  and  Accredita¬ 
tion  become  less  time  consuming  and 
expensive. 

Conclusions 

The  objective  of  the  Army’s  M&S 
Standards  Development  Process  is  to  cre¬ 
ate  an  environment  that  promotes  the  shar¬ 
ing  and  reuse  of  M&S  procedures,  prac¬ 
tices,  processes,  techniques,  algorithms  and 


Define 

Requirements 


Define  sUiidirds  required 
Develop  forums 


>  Identify  Experts  ■■  DoD, 
industry,  academia 

•  Devdop  fonuns 

*  Exchange  information  and  ideas 


Done  concurrently  with  other  steps 
Goal:  Embed  standards  within  community 


Standards  Qb|cctives 
Interoperability  Commonality 

Reuse  Credibility 


FIGURE  2.  The  Army’s  M&S  Standards  Development  Process 


heuristics.  It  is  not  intended  to  create  a 
set  of  written  standards  that  will  sit 
unused  in  ASTARS,  The  continuous  and 
iterative  nature  of  the  process  permits  the 
Army  M&S  community  to  keep  pace 
with,  rather  than  react  to,  technological 
advances.  Via  SNAP  and  ASTARS,  the 
Army  moves  progressively  closer  to  its 
goal  of  developing  “a  comprehensive  set 
of  standards  that  facilitates  efficient 
development  and  use  of  M&S  capabili¬ 
ties”  and  reduces  the  gap  between  the 
current  and  the  desired  state. 


I 

i  By  using  approved  standards, 
M&S  developers  have  a  solid 
foundation  upon  which  to  build. 

The  developers  can  begin  their  design 
efforts  by  querying  ASTARS 
to  see  if  a  standard  presently  exists 
to  meet  their  needs. 

Resources 

DoD  5000. 59-P;  DoD  Modeling  and  Sim¬ 
ulation  Master  Plan,  October  1995 
Army  Model  and  Simulation  Standards 
Report  FY  98,  October  1997 


The  Army  Model  and  Simulation  Master 
Plan,  October  1997 
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Alternative  Measures  of  Performance  for  Testing 
Command  and  Control  Systems 
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Joint 

Interoperability 
Test  Command 


The  Joint  Interoperability  Test  Com¬ 
mand  (JITC)  recently  completed 
Operational  Test  (OT)  for  the 
Global  Command  and  Control  System 
(GCCS)  Version  3.0,  the  current  Depart¬ 
ment  of  Defense  Command  and  Control 
System  of  Record.  To  achieve  testing 
economies  and  to  overcome  specific  chal¬ 
lenges,  JITC  implemented  surrogate  Mea¬ 
sures  of  Performance  (MOPs)  that  have 
broad  applicability  to  other  command,  con¬ 
trol,  communications,  computers  and  intel¬ 
ligence  (C4I)  systems. 

Surrogate  Measures  for  Reliability, 
Availability  and  Maintainability 
(RAM) 

Traditional  measures  of  RAM  are 
Mean-Time-Between-Operational-Mis- 
sion-Failures  (MTBOMF),  Operational 
Availability  (A^),  and  mean-time-to-repair 
(MTTR).  All  of  these  traditional  measures 
require  a  strict  dichotomy  between  system 
operating  time  and  system  down  time;  in 
other  words,  they  require  the  system  to  be 
either  “up”  (operating)  or  “down”  (not 
operating). 

In  the  highly  distributed,  client-server, 
multiprocessor  environment  of  GCCS,  the 
dichotomy  between  operating  time  and 
down  time  does  not  exist.  Instead,  many 
levels  of  degraded  operation  exist  between 
up  and  down.  Further,  the  characterization 
of  the  “system”  is  blurred  because  the  sys¬ 
tem  may  be  defined  as: 

•  The  entire  network  of  over  600  sites. 


•  A  single  site,  many  of  which  have 
unique  configurations. 

•  A  user  workstation,  of  which  there 
are  three  distinct  types  and  several 
unique  configurations. 

As  a  result,  the  GCCS  community 
selected  alternative  characteristics  in  lieu  of 
the  traditional  RAM  measures.  Specifical¬ 
ly,  the  user  community  emphasized 
requirements  for  accessibility,  consistency 
and  dependability. 

Accessibility  is  the  ability  of  users  to 
log  on  and  begin  executing  processes  in 
support  of  mission  operations.  Test  mea¬ 
sures  were  time  to  log  on  and  time  to 
access  applications.  The  assessment  con¬ 
sidered  both  local  and  remote  (dial-up) 
access;  it  also  considered  multiple  worksta¬ 
tion  types  and  multiple  applications.  The 
analysis  focused  on  timeliness  in  opera¬ 
tional  terms  (in  other  words,  can  the  user 
access  the  system  in  time  to  support  mis¬ 
sion  accomplishment). 

Consistency  is  the  ability  of  the  system 
to  provide  the  same  results  to  a  given 
process  regardless  of  where  it  is  initiated 
and,  if  the  underlying  data  has  not 
changed,  regardless  of  when  it  is  initiated. 
The  test  measure  was  comparison  of  sys¬ 
tem  output  for  identical  processes.  To 
ensure  the  test  addressed  user  concerns,  the 
test  focused  on  specific  applications  nomi¬ 
nated  by  the  user  community. 

Dependability  is  the  ability  of  the  sys¬ 
tem  to  complete  user-initiated  processes  to 
the  user’s  satisfaction.  Test  measures  were 
frequency  and  duration  of  workstation  out¬ 
ages.  These  measures  focused  on  the  abili¬ 
ty  of  the  user  to  complete  assigned  mission 
tasks.  The  analysis  considered  three  types 
of  outages:  corrected  by  the  operator,  cor¬ 
rected  by  site  personnel  other  than  the 
operator  and  corrected  by  off-site  person¬ 
nel.  This  approach  was  selected  to  facili¬ 
tate  analysis  of  any  supportability  short¬ 
falls. 

Table  1  provides  a  summary  compari¬ 
son  of  the  traditional  and  surrogate  mea¬ 
sures. 

The  overall  benefits  of  these  surrogate 
measures  were: 

•  The  test  command  avoided  the  pitfalls 


associated  with  traditional  RAM  mea¬ 
sures  used  in  previous  tests. 

•  The  test  focused  on  issues  of  primary 
concern  to  the  user. 

•  Test  results  provided  a  baseline  for  com¬ 
parison  to  similar  results  for  future  ver¬ 
sions. 

Summary.  The  traditional  RAM  mea¬ 
sures  may  not  apply  in  complex  distributed 
systems.  In  such  cases,  the  test  community 
should  be  open  to  applying  meaningful 
surrogate  measures  that  support  the  evalua¬ 
tion  from  the  user  perspective.  Accessibili¬ 
ty,  consistency  and  dependability  are  valid 
candidate  measures  and  should  be  consid¬ 
ered  in  future  tests. 

Measuring  the  Ability  of  the  System 
to  Support  User  Missions 

The  test  command  was  faced  with  two 
significant  challenges  in  the  operational 
evaluation: 

•  criteria.  For  the  most  part,  the 
user-defined  criteria  were  subjective  and 
did  not  have  well-defined  quantitative 
thresholds. 

•  Comparison  to  previous  version.  The 
fundamental  requirement  for  fielding 
Version  3.0  was  to  match  the  functionali¬ 
ty  of  the  previous  version.  However,  a 
quantitative  performance  baseline  for  the 
previous  version  did  not  exist. 

To  meet  this  challenge,  the  testers,  in 
concert  with  the  user  community,  estab¬ 
lished  a  test  measure  to  assess  the  ability  of 
the  system  to  support  user  missions.  This 
broad-based  measure,  known  as  Subject 
Matter  Expert  (SME)  assessment  of  mis¬ 
sion  tasks,  supported  evaluation  of  three  of 
the  five  critical  operational  issues.  As  a 
result,  the  TEMP  identified  it  as  the  prima¬ 
ry  measure  of  performance  for  the  system. 

Figure  1  illustrates  the  approach  for 
incorporating  the  SME  assessment 
throughout  all  phases  of  the  test  and  evalu¬ 
ation  process:  test  planning,  test  execution 
and  test  analysis  and  reporting. 

The  user  community  participated  in 
developing  the  mission  task  list,  selecting 
those  tasks  that  were  most  critical.  Three 
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Table  1.  Comparison  of  Traditional  and  Surrogate  Measures 


Traditional  Measures 

Surrogate  Measures 

Reliability 

Accessibility 

MTBOMF  =  #  of  operating  hours  / 

Time  to  log  on  =  Elapsed  time  from  “Password  Entry” 

#  of  operational  mission  failures 

to  “Desktop  Ready”  (System  Ready  to  Respond  to 

User  Requests) 

AvailabUity 

Aq  =  #  of  operating  hours  /  (#  of 
operating  hours  +  #  of  down  hours) 

Time  to  access  applications  =  Elapsed  time  from 
“Icon  Click”  to  “Application  Window  Ready” 

Consistency 

Maintainability 

MTTR  =  #  of  hours  in  repair  / 

Manual  comparison  of  printed  output  reports 

#  of  repair  actions 

Dependability 

Frequency  of  workstation  outages  =  #  of  outages  per  day 

Duration  of  workstation  outages  =  elapsed  time  from  “Out¬ 
age  (Lockup)  Perceived  by  User”  to  “Functionality 
Restored” 

types  of  mission  tasks  were  assessed: 

•  User  mission  task.  A  staff  action  that 
requires  system  support  to  complete;  it 
has  defined  product  and  takes  about  one 
to  four  hours  to  complete.  Examples: 

•  Develop  and  enter  common-user  air 
movement  schedules. 

•  Produce  a  non-combatant  evacuation 
list  for  the  planned  area  of  interest. 

•  Produce  an  intelligence  resources 
report. 

•  Installation  mission  task.  A  function 
performed  by  a  system  administrator  or 
other  site  personnel  to  install  and  config¬ 
ure  the  system  in  preparation  for  sus¬ 
tained  operations.  Examples: 

•  Install  and  de-install  software  seg¬ 
ments. 

•  Set  up  and  maintain  user  accounts. 

•  Establish  local  site  configuration. 

•  Sustained  operations  mission  task.  A 
function  performed  by  site  personnel  to 
support  sustained  operations  of  the  sys¬ 
tem.  Examples: 

•  Access  audit  capabilities. 

•  Estabhsh  and  maintain  database  struc¬ 
ture. 

•  Administer  and  monitor  the  network. 

For  user  mission  tasks,  the  SME  was  a 
system  user  at  the  test  site  (for  example,  a 
staff  officer  who  uses  the  system  products 
to  support  mission  accomplishment);  in 


general,  the  mission  task  SME  was  not  the 
workstation  operator  who  performed  the 
task.  For  installation  and  sustained  opera¬ 
tions  mission  tasks,  the  SME  was  a  mem¬ 
ber  of  the  GCCS  team  at  the  test  site  who 
was  sufficiently  knowledgeable  in  the 
function  to  make  the  assessment. 

SMEs  observed  and  assessed  each  task. 
For  user  mission  tasks,  the  assessment  was 
based  on  the  timeliness,  accuracy,  com¬ 
pleteness  and  usefulness  of  the  mission 
task  output/product  (in  other  words,  on  the 


“goodness”  of  the  information  provided  by 
the  system).  SMEs  used  this  rating  system 
to  assess  the  results: 

•  Fully  successful  Product  is  sufficiently 
timely,  accurate,  complete,  and  useful. 
Fully  supports  mission  accomplishment 
by  the  intended  recipient. 

•  Marginally  successful  Product  is  accu¬ 
rate  and  timely;  it  supports  mission 
accomplishment.  However,  the  user  is 
required  to  perform  workarounds  to  gen¬ 
erate  or  use  the  product. 

•  Unsuccessful  No  product,  product  can¬ 
not  be  used  by  the  intended  recipient  for 
the  intended  purpose,  or  the  product  is 
too  late  or  inaccurate  to  support  the  mis¬ 
sion. 

For  installation  and  sustained  operations 
tasks,  SMEs  assessed  both  the  success  and 
the  ease  of  the  task  considering  training, 
documentation,  timeliness  and  other  fac¬ 
tors. 

The  benefits  of  this  assessment  approach 
were: 

•  Challenges  associated  with  “soft” 
requirements  and  the  lack  of  quantitative 
thresholds  were  overcome. 

•  The  user  community  was  involved  early  in 
the  test  process,  ensuring  the  evaluation 

(5^^  TESTING,/?.  35) 


Test  Planning  -  User  establishes  ^ 
scenario,  mission  tasks  and  criteria. 


Test  Conduct  -  Test  participants 
accomplish  tasks.  Subject  matter 


Evaluation  -  Compute  percentage  of 
mission  task  success.  Assess 
operational  impact  of  non-success. 


Example 


Mission  Task  Success 


Fully  Marginally  Unsuccessful 
Successful  Successful 


Assess  operatioial  impact  of 
I  non-successes: 

Minimal  -  users  experience 
inconvenience 
Moderate  -  users  experience 
delays 

Significant  -  users  are  unable  to 

accomplish  key  mission 


Figure  1.  Measuring  Mission  Task  Success 
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Building  Complexity  into  Biomedical  Models 


Introduction 

Because  human 
response 
Input-  Output 
(I/O)  relationships 
seem  to  be  tremen¬ 
dously  complex,  not 
easily  amenable  to 
neat  analytic  solu¬ 
tions,  and  include  sto¬ 
chastic  and  chaotic 
characteristics,  they  are  difficult  to  compre¬ 
hensively  model  and  simulate  with  high 
degrees  of  fidelity  and  predictive  accuracy. 
Nevertheless,  relatively  simple,  opera¬ 
tionally  useful,  first-order  models  of  soldier 
responses  to  environmental  stressors  and 
survivability  threats  have  been  implement¬ 
ed  and  integrated  into  military  applications. 
The  seemingly  intractable  complexities  of 
the  human  system  have  been  successfully 
managed  by  various  complexity  reduction 
techniques  such  as  abstraction,  domain 
restriction,  simplifying  assumptions, 
effects  aggregation,  order  reduction, 
lookup  tables  and  mles-of-thumb. 

Simplified  soldier  response  models  are 
a  practical  necessity  for  many  apphcations 
but  they  can  also  serve  as  platforms  for  the 
development  of  models  having  greater 
complexity  and  realism.  Complexity  can 
be  added  to  increase  a  model’s  scope  and 
usability,  explore  second  and  higher-order 
effects  and  simulate  anticipatory  and 
delayed  responses,  feedback  modulation, 
and  other  important  intrinsic  human  capa¬ 
bilities  such  as  noise  filtering,  prediction 
and  pattern  recognition. 

Elements  of  Complexity 

There  are  numerous  aspects  to  com¬ 
plexity  in  modeling  the  human  element. 
Here  we  review  some  of  the  latent  com¬ 
plexities  in  basic  physiological  and  cogni¬ 
tive  system  transfer  functions.^  There  are 
additional  sources  of  human  complexity 
such  as  multitasking,  communications, 
decision  making  and  motivation  that  are 
not  discussed  in  this  paper.^’  ^ 

Number  of  Variables 

Simple  models,  almost  by  definition, 
have  few  predictor  and  response  variables. 
Conversely,  the  greater  the  number  of  vari¬ 


ables,  the  more  complex  the  model.  How¬ 
ever,  as  multistep  regression  analysis  often 
demonstrates,  adding  more  independent 
variables  alone  does  not  always  signifi¬ 
cantly  improve  a  model’s  predictive  capa¬ 
bility. 

Additional  dependent  variables  can 
expand  the  scope  of  a  model’s  usability 
but  often  requires  a  concomitant  increase 
in  the  number  of  input  variables  and  does 
not  itself  improve  a  model’s  resolution. 

Non-linearity  and  Interaction 

The  characterization  of  human  system 
responses  ranges  from  almost  linear  to 
highly  nonlinear.  Although,  for  many 
applications,  nonlinear  responses  can  be 
adequately  approximated  using  piecewise- 
continuous  linear  I/O  models,  high  fidelity 
representations  may  require  implementa¬ 
tion  of  more  accurate  and  realistic  nonlin¬ 
ear  system  descriptions. 

Nonlinear  modeling  terms  are  not  as 
problematic  from  the  algorithmic  view¬ 
point  as  they  are  when  included  as  compo¬ 
nents  of  dynamic-response  models.  In  such 
instances,  they  can  make  obtaining  closed- 
form  solutions  very  difficult  (requiring  an 
expert  in  applied  mathematical  methods) 
or  impossible.  Nonetheless,  even  simple 
soldier  response  prediction  algorithms 
often  include  nonhnear  terms.  Perhaps  the 
most  frequent  nonhnear  representations  are 
exponential  functions  of  input  or  interme¬ 
diate  process  (state)  variables. 

Compartment  models  are  frequently 
used  to  model  dynamic  physiological 
processes  and  are  therefore  typically  for¬ 
mulated  as  systems  of  coupled  hnear  dif¬ 
ferential  equations."^  These  are  relatively 
straightforward  to  interpret  and  implement. 
A  difficulty,  with  adding  nonlinear  terms 
to  such  models,  however,  is  that  the  result¬ 
ing  system  of  equations  will  often  become 
analytically  intractable.  However,  a 
numerical  solution  can  be  obtained  using 
software  algorithms  (often  very  complex 
when  optimized  for  robust  manipulation  of 
ill-conditioned  matrices).  A  significant 
disadvantage  of  not  being  able  to  derive  a 
closed-form  solution  is  that  results  from 
numerical  methods  cannot  be  directly  veri¬ 
fied. 

Another  point  to  consider  is  that  nonhn- 
earities  due  to  use  of  higher  powers  of 


independent  variables  can  improve  predic¬ 
tive  precision  to  a  certain  degree.  In  many 
cases,  however,  a  lower-power  cross-prod¬ 
uct  term  will  have  greater  effect  in  improv¬ 
ing  a  model’s  predictive  capability.  First- 
order  cross  products  can  also  be  interpreted 
as  representing  the  effects  of  variable,  or 
factor,  interactions.  The  meaning  of  high¬ 
er-power  terms  is  usually  more  difficult  to 
discern. 

Adding  nonlinear  terms  to  an  estab- 
hshed  algorithm  is  usually  trivial,  in  itself. 
However,  it  may  be  costly  due  to  the  need 
to  re-determine  coefficient  values  to  prop¬ 
erly  apportion  linear,  power,  and  interac¬ 
tion  effects.  Also,  robust  coefficient  deter¬ 
minations  for  models  with  interaction 
terms  can  only  be  assured  if  data  used  for 
coefficient  identification  were  derived 
from  studies  exphcitly  designed  to  evaluate 
interaction  effects  in  statistically  valid 
ways. 

Time  Invariance 

The  human  I/O  processing  mechanism 
is  usually  not  time  invariant  because 
responses  to  inputs  typically  differ  as  func¬ 
tions  of  time  of  application  of  stimuli. 
Training  and  habituation  make  many  some 
input/output  relationships  approximately 
time  invariant,  however,  other  responses 
remain  very  time  dependent.  In  many 
cases,  however,  time  is  actually  a  surrogate 
for  number  of  previous  exposures  (experi¬ 
ence  factor).  That  is,  the  coefficients  for  the 
predictor  variables  are  functions  of  time  or 
frequency  with  which  a  particular  input 
profile  has  been  applied. 

Causality 

Causal  systems  generate  outputs  based 
only  on  current,  or  previous,  but  not  future, 
or  anticipated,  inputs.  Because  humans  nat¬ 
urally  formulate  and  use  mental  models  to 
facilitate  estimation,  prediction  and  deci¬ 
sion  making,  situational  responses  are 
functions  of  not  only  present  and  past 
exposures  but  also  of  predicted  future 
events.  In  this  sense,  human  system 
responses  involving  cognitive  processing 
are  not  always  causal.  This  is  an  additional 
source  of  complexity  that  can  be  addressed 
when  striving  to  develop  accurate  soldier 
response  and  decision  making  models. 


Dr.  Matthew 
Reardon 
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Feedback 

Systems  often  contain  or  are  part  of 
feedback  mechanisms.  The  nature  of  feed¬ 
back  (positive  or  negative)  determines  the 
nature  of  system  responses  to  both  deter¬ 
ministic  inputs  and  random  perturbations. 
Feedback  response  characteristics  (e.g., 
damped  or  undamped)  are  determined  by 
constitutive  properties  incorporated  mathe¬ 
matically  into  coefficients  and  parameters. 
A  soldier,  for  example,  utilizes  direct  sen¬ 
sory  and  model-based  feedback  to  direct 
and  adjust  behavior  and  achieve  objectives 
and  specified  levels  of  performance  despite 
changes  in  internal  parameters.  Hence, 
closed  loop  soldier  models  are  preferred 
over  their  open  loop  alternatives  for  high 
fidelity,  constructive  soldier-based  simula¬ 
tions. 

Uncertainty 

Human  response  models  that  only  use 
mean  values  for  coefficients  and  parame¬ 
ters  are  effectively  deterministic.  Associat¬ 
ed  second  order  statistics  for  these  values 
are  required  to  create  stochastic  human 
models  that  can  be  used  for  Monte  Carlo 
simulation.  Surprisingly  few  models  pro¬ 
vided  such  data,  unnecessarily  limiting 
their  application.  The  assumption  regard¬ 
ing  parameter  variability  is  that  the  distrib¬ 
ution  is  Gaussian  (therefore  obviating  the 
need  for  higher-order  cummulants)  and 
that  the  distributions  do  not  vary  as  func¬ 
tions  of  time  or  other  variables  (i.e.;  the 
statistics  are  stationary).  It  is  doubtful  that 
most  human  processes  are  exactly  station¬ 
ary.  Therefore  higher  fidelity  will  occur 
with  nonstationary  models. 

Additionally,  human  responses  to  iden¬ 
tical  sequences  of  conditions  and  events 
typically  vary  considerably  between  indi¬ 
viduals.  Sources  of  such  interpersonal  vari¬ 
ances  include  myriad  differences  between 
individuals  in  basic  characteristics  and 
experience  as  well  as  differences  in  sensi¬ 
tivity  to  changes  in  external  and  internal 
variables  on  cognitive,  affective  and  physi¬ 
ological  responses. 

Optimization 

Humans  act  instinctively  in  attempts  to 
optimize  state  trajectories  based  on  qualita¬ 
tive  minimization  or  maximization  of  vari¬ 
ous  subjective  functions  of  sensations  and 
potential  consequences.  For  example 
behavior  patterns  are  directed  toward  mini¬ 
mizing  effort,  energy,  task  completion 


time,  distance,  expense,  inconvenience, 
stress,  distress,  etc.  Alternatively,  behavior 
is  directed  toward  maximizing  conve¬ 
nience,  comfort,  recognition,  awards, 
recreational  time,  etc.  Behavior,  however, 
is  usually  driven  by  efforts  to  simultane¬ 
ously  minimize  adverse  consequences  and 
maximize  beneficial  ones. 

Software-implemented  soldier  models 
typically  act  deterministically  according  to 
scripts  or  prescribed  rules.  However,  real 
soldiers  are  more  complex  and  tend  to  act 
in  ways  that  minimizes  effort  and  exposure 
to  stressors.  Humans  attempt  to  efficiently 
reach  objectives  by  qualitatively  optimiz¬ 
ing  a  subjectively  weighted  sum  of  gain 
and  cost  functions.  We  know,  however, 
that  this  process  does  not  always  work  that 
well.  There  may  be  considerable  inter-per¬ 
sonal  variability  regarding  selected  opti¬ 
mization  functions  as  well  as  variability  in 
their  relative  weightings. 

Noise  Filtering 

Humans  routinely  filter  sensory  input  to 
reduce  the  effects  of  noise  and  improve  the 
effective  signal  to  noise.  Therefore  human 
response  models  can  be  made  more  realis¬ 
tic  by  appropriately  including  this  capabili¬ 
ty.  Filtering  noisy  input  improves  dynamic 
performance. 


Verification  and  Validation 

Verification  of  complex  human  models 
in  many  cases  will  not  be  possible  since  the 
theoretical  or  mathematical  foundations  for 
much  of  the  nonphysiological  components 
of  human  models  are  inductive  rather  than 
deductive.  Hence  the  best  that  can  usually 
be  done  is  to  draw  upon  validated  methods 
or  forms  and  validate  the  performance  of 
the  subsequent  model.  Unfortunately,  how¬ 
ever,  adding  even  a  small  amount  of  com¬ 
plexity  to  a  model  may  make  it  impossible 
to  effectively  validate.  This  can  occur,  for 
example,  if  a  model’s  dimensionality 
requires  a  combinatorially  large  number  of 
studies  to  span  the  operationally  required 
vahdation  envelope. 

These  problems  do  not  obviate  the  use¬ 
fulness  of  complex  models.  They  can  stiU 
be  useful  for  testing  concepts  and  generat¬ 
ing  hypotheses  that  can  be  specifically  test¬ 
ed  in  focused  studies.  Such  models  can 
also  be  used  to  evaluate  sensitivity  of 
human  system  responses  to  changes  in 
characteristics  (coefficients  and  parame¬ 
ters),  defining  the  statistical  characteristics 
of  model  outputs,  and  as  a  basis  for  revert¬ 
ing  to  an  extended  simplified  model  using 
complexity  reduction  techniques  such  as 
linearization. 

(See  BIOMEDICAL  MODELS,/?.  32) 


Simplification  byr 


Figure  1  -  Building  Realistic  Human  Performance  Models  by 
Adding  Complexity  to  Simple  Models 
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Working  Groups/Composite  Groups  and  the  67th  MORSS 

CDR  Kirk  Michealson,  WG/CG  Coordinator 
MAJ  Steve  Horton,  Deputy  Site  Coordinator 


The  67th  MORSS  will  be  held  22> 
24  June  1999  at  the  United  States 
Military  Academy,  West  Point, 
NY,  with  a  theme  of  Focusing  Military 
Operations  Research:  From  our  Her¬ 
itage  to  the  Future.  The  Announcement 
and  Call  for  Papers  (ACP)  has  already 
been  sent  out,  and  you  should  have  your 
copy  with  the  details  for  the  Symposium. 
The  ACP  has  additional  information  on 
preparing  abstracts,  contact  information 
of  the  Composite  Group  and  Working 
Group  Chairs,  and  discussion  on  areas  of 
interests  for  each  of  the  working  groups. 
If  you  haven’t  received  your  ACP  yet, 
please  call  Cynthia  Kee-LaFreniere  or 
Christine  Parnell  in  the  MORS  Office 
to  request  one  [(703)  751-7290],  or  you 
can  check  the  67th  MORSS  link  from  the 
MORS  home  page  (http://www. 
mors.orgO. 

Abstracts  for  all  proposed  presenta¬ 
tions  are  due  by  22  January  1999.  Send 
yours  directly  to  the  Working  Group  or 
Composite  Group  Chair  you  believe 
would  be  most  interested,  or  to  the 
MORS  Office,  if  you  are  uncertain,  and 
they  will  forward  it  to  the  appropriate 
Chair.  The  quality  of  papers  determines 
the  overall  success  of  the  Symposium. 
Do  not  hesitate  to  present  a  paper.  A 
fully-developed  paper  is  not  required,  just 
a  presentation  pitch.  Also,  the  research 
does  not  need  to  be  completed  or  a  con¬ 
clusion  reached,  we  encourage  works  in 
progress  —  peer  critiques  are  excellent! 
This  is  a  good  way  to  receive  “free”  sug¬ 
gestions  from  your  peers  on  how  your 
analysis  is  going  and  how  you  can  make 
it  better. 

Like  the  66th  MORSS,  the  Composite 
Group  and  Working  Group  Chairs  will 
again  have  eight  group  sessions,  all  in  the 
0830-1000,  1030-1200  and  1330-1500 
timeslots  (except  for  the  0830-1000 
timeslot  on  Tuesday  when  the  Plenary 
Session  is  scheduled).  The  Special  Ses¬ 
sions  are  also  in  the  same  timeframe  as 
last  year,  from  1530  to  1700  daily.  To 


Figure  1.  Thayer  Hall 


Figure  2.  Typical  Thayer  Hall  Classroom 
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67th  MORS  SYMPOSIUM 


Keynote  Speaker  Confirmed! 

General  John  N.  Abrams 

Commanding  General, 

United  States  Army  Training  and  Doctrine  Command, 
has  agreed  to  be  the  Keynote  Speaker 
at  the  67th  MORS  Symposium,  USMA 
23-25  June  1999 


view  a  summary  agenda/  session  sched¬ 
ule  for  this  year’s  Symposium,  please 
look  at  the  67th  MORSS  link  on  the 
MORS  home  page. 

From  the  WG/CG  perspective,  we  are 
looking  forward  to  this  year’s  Sympo¬ 
sium  at  West  Point.  The  classroom  facil¬ 
ities  there  are  ideal  for  this  purpose.  Not 
only  will  almost  every  WG  and  CG  ses¬ 
sion  be  conducted  on  one  floor  of  one 
building,  but  the  rooms  themselves  are  all 
outfitted  with  powerful  computer  equip¬ 
ment  and  software. 

The  WG  and  CG  sessions  for  the  67th 
MORSS  will  be  held  in  Thayer  Hall  (see 
Figure  1).  Most  classrooms  in  Thayer 
Hall  are  configured  to  seat  20  cadets  at 
small  student  desks.  While  there  are 
some  larger  rooms  available  for  the  CG 
sessions  and  for  some  of  the  larger  WGs, 
many  of  the  WG  sessions  will  be  con¬ 
ducted  in  these  20  person  rooms.  We 
intend  to  add  a  few  chairs  and  remove 
enough  desks  to  boost  the  capacity  of 
these  rooms  for  MORSS  to  30  or  so.  Fig¬ 
ure  2  shows  a  typical  Thayer  Hall  class¬ 
room  in  use. 

Every  WG  and  CG  presentation  room 
contains  a  Pentium  90/100  PC  with  mul¬ 
timedia  that  is  connected  to  the  network. 
Currently,  these  PCs  operate  with 
Microsoft  NT  4.0.  The  software  load 


includes  Microsoft  Office  ’97,  so  Power¬ 
Point  presentations  will  be  easy  to  han¬ 
dle.  In  addition,  these  rooms  all  have  an 
Epson  projection  device  that  projects  a 
large  and  clear  image  of  the  computer 
monitor  on  to  a  screen.  A  hand-held 
remote  control  allows  the  briefer  to  per¬ 
form  aU  mouse  functions  from  anywhere 
in  the  room.  This  system  is  easy  to  learn 
and  is  a  very  effective  means  of  present¬ 
ing  information.  There  is  also  a  VHS  for¬ 
mat  VCR  that  runs  through  the  projection 
device  available  in  each  room.  Finally, 
each  room  has  a  sturdy  and  well-used 


overhead  projector  that  can  be  used  by 
anyone  so  inclined.  Figure  3  shows  the 
presentation  system  in  operation. 

West  Point  intends  to  provide  elec¬ 
tronic  storage  space  for  unclassified  Pow¬ 
erPoint  presentations  that  presenters  want 
to  e-mail  to  the  site,  and  they  also  plan  to 
have  a  small  number  of  computers  avail¬ 
able  for  presenters  who  wish  to  update 
their  presentations  during  the  Sympo¬ 
sium.  The  details  of  these  procedures 
will  be  published  later.  None  of  the  com¬ 
puters  are  certified  for  classified  opera¬ 
tion,  so  classified  presentations  must 
either  be  in  hardcopy  form  (either  mail 
them  or  hand  carry  them)  or  on  the  pre¬ 
senter’s  own  (certified)  laptop.  Please  do 
not  attempt  to  e-mail  any  classified  mate¬ 
rial  to  West  Point.  Details  regarding  clas¬ 
sified  presentations  will  be  published  at  a 
later  date. 

Finally,  West  Point  intends  to  make 
extensive  use  of  the  internet  to  dissemi¬ 
nate  information.  General  information 
about  USMA  is  available  at 
http://www.usma.edu/.  General  informa¬ 
tion  about  the  67th  MORSS  is  available 
at  http://www.mors.org/.  The  site  com¬ 
mittee  stands  ready  to  do  their  part  to 
make  the  67th  MORSS  the  best  ever. 

If  you  have  any  doubts  at  all  about 
presenting  a  paper,  have  any  other  ques¬ 
tions,  or  have  not  received  the  ACP, 
please  contact  the  MORS  Office  at  (703) 
751-7290  or  the  WG/CG  Coordinator, 
CDR  Kirk  Michealson,  at  (703)  697- 
0064.  O 


Figure  3.  Classroom  Presentation  Equipment 
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Is  “Non-Selection”  the  Same  as  “Elimination?” 


Abstract 

I  propose  a  rough 
model  that  sug¬ 
gests  there  is  a 
significant  difference 
between  non-selec¬ 
tion  and  elimination 
as  typically  practiced 
by  “selection”  com¬ 
mittees  and  other 
choice  processes.  I 
argue  that  Army  offi¬ 
cer  selection  boards 
appear  to  be  operating  as  elimination 
processes.  This  has  interesting  implica¬ 
tions  for  individual  officers  aspiring  to 
higher  ranks,  and  suggests  this  leads  to  a 
corps  of  retained  officers  having  certain 
qualities.  Applying  the  elimination 
process  in  the  succession  of  boards 
involved  in  selecting  officers  for  increas¬ 
ing  ranks  amounts  to  a  sequence  of  suc¬ 
cessive  (“compounding”)  truncations  of  a 
parent  distribution.  This  contributes  to 
having  a  set  of  senior  officers  whose  rat¬ 
ings  are  sometimes  said  to  be  “nearly 
indistinguishable.” 

Committees  formed  to  “select”  the 
most  attractive  from  a  group  of  candidates 
in  many  cases  actually  function  by  elimi¬ 
nating  the  non-attractive  candidates.  I 
shall  argue  that  the  sets  of  candidates  cho¬ 
sen  by  these  processes  are  quite  different. 
For  example,  the  committee  formed  in  a 
university  mathematics  department  to 
select  the  calculus  textbook  to  be  adopted 
for  next  year’s  classes  typically  begins  by 
looking  for  objectionable  attributes  in  the 
candidate  texts.  Any  text  found  to  have  an 
undesirable  (or  even  unusual)  quality  is 
usually  eliminated  from  further  considera¬ 
tion.  What  remain  are  candidates  having 
no  unusual  characteristics;  the  elimination 
process  tends  to  generate  finalists  that  are 
similar  and  “middle  of  the  road”  in  every 
respect  considered.  In  addition  to  text¬ 
book  “selection”  committees,  examples  of 
elimination  processes  include  personnel 
departments  recruiting  to  fill  mid-level 
openings  in  a  corporation  and  how  we 
select  a  rental  video  for  viewing  by  the 
family. 

A  crude  model  can  be  stated  in  terms 
of  the  score  Tjj  of  candidate  i  in  dimen¬ 
sion  (or  attribute)  j,  where  we  agree  to 


define  scores  so  that  “bigger  is  better”  in 
all  cases.  The  set  E  of  survivors  of  the 
elimination  process  is,  roughly, 
£  =  {:V7,r„  3  g,}  where  gj  is  the 
lower  bound  of  good  performance  in 
dimension  j.  For  an  individual  to  remain 
in  the  set  of  survivors,  he  or  she  must  have 
no  instances  of  less  than  good  perfor¬ 
mance  in  any  dimension. 

Selection  processes,  on  the  other  hand, 
tend  to  seek  outstanding  (and  sometimes 
unusual)  qualities  and  even  to  trade  these 
off  against  mild  shortcomings.  The  selec- 


Coiranittees  formed  to  “select” 
the  most  attractive  from  a 


group  of  candidates  in  many  cases 
actually  function  by  elimimting  the 
non-attractive  candidates. 

tion  process  tends  to  generate  finalists  that 
are  quite  different.  Every  day  examples  of 
selection  processes  include  sports  “win¬ 
ners”,  such  as  horses  showing  in  a  day  of 
racing  at  a  given  track,  or  selection  of  a 
mutual  fund  based  on  performance  over 
the  past  year.  Perhaps  Darwin’s  natural 
selection  is  truly  a  selection  process.  As 
yet  another  example  of  selection,  the  stu¬ 
dent  admission  process  here  at  the  US 
Military  Academy  evaluates  candidates  in 
areas  such  as  grades,  SAT  scores,  leader¬ 
ship,  sports  and  service.  Admissions  offi¬ 
cers  tend  to  look  favorably  upon  a  candi¬ 
date  having  outstanding  qualifications  in 
one  or  more  of  these  areas,  even  though 
he  or  she  does  not  score  exceptionally 
high  in  other  areas.  Indeed,  there  is  a 
deliberate  effort  to  admit  candidates  so  as 
to  form  a  corps  of  “qualified”  cadets  hav¬ 
ing  a  wide  variety  of  attributes.  (Admit¬ 
tedly,  there  is  an  initial  elimination  of  can¬ 
didates  whose  SAT  scores  are  below  some 
threshold.  Thus  the  overall  process  is 


actually  elimination,  followed  by  selec¬ 
tion.) 

In  the  selection  cases,  there  may  well 
have  been  moderate  (or  even  poor)  per¬ 
formance  in  various  dimensions,  or  at 
other  times,  but  spectacular  performance 
in  a  given  dimension  over  the  period  con¬ 
sidered  may  lead  to  selection.  In  terms  of 
our  model,  the  set  S  of  survivors  in  a 
selection  process  may  be  something  like 
S  >Oj}  where  Oj  is  a  bound 

defining  outstanding  performance  in 
dimension  j. 

It  is  clear  from  their  defining  condi¬ 
tions  that  E  and  S  are  quite  different.  To 
illustrate,  suppose  there  are  three  candi¬ 
dates  being  assessed  with  respect  to  two 
attributes.  Let  us  identify  candidate  1  by 
his  vector  of  scores  (T^^,  T22)  =  (5,  4), 
and  similarly  for  candidates  2  and  3  with 
scores  (2,  17)  and  (8,  2),  respectively. 
Suppose  gj  =  g2  =  3  and  0^  =  02=  10. 
With  an  elimination  process,  candidate  1 
would  be  the  sole  survivor  (i.e.,  the  only 
member  of  E),  whereas  candidate  2 
would  be  the  only  member  of  the  selected 
set  S. 

By  our  crude  model,  the  process  of 
selecting  the  candidates  to  be  excluded 
(i.e.,  “selecting  out”)  gives  an  excluded 
set  defined  by  O  <  g  j\  . 

Since  negation  of  the  condition  defining 
O  gives  the  condition  defining  E,  it  fol¬ 
lows  that  O  and  E  (the  complement  of 
E)  are  the  same.  That  is,  eliminating  can¬ 
didates  is  equivalent  to  selecting  them 
out.  It  is  therefore  somewhat  surprising 
that  the  sets  of  retained  candidates  result¬ 
ing  from  elimination  and  selection  are  so 
different.  (Interestingly,  when  only  one 
attribute  is  under  consideration,  defini¬ 
tions  of  S  and  E  differ  only  in  the  values 
of  the  thresholds  o  and  g.  Presumably  in 
some  situations  these  would  be  made  to 
coincide,  in  this  case,  in  order  for  the 
processes  to  generate  the  desired  yield  of 
retained  candidates.) 

I  believe  promotion  boards  for  officers 
in  the  US  Army  appear,  as  a  first  approxi¬ 
mation,  to  be  operating  more  like  elimina¬ 
tion  boards  than  selection  boards,  partly 
in  response  to  the  downsizing  effort.^ 
This  has  clear  implications  for  both  the 
officers  aspiring  to  higher  ranks  and  the 
nature  of  the  officer  corps.  For  the  indi- 


Dr.  Donald 
Barr 

US  Military 
Academy 


PHALANX 


20 


December  1998 


Figure  1,  Plot  of  the  density  of  a  N(0,1)  population  truncated  below  at  5.0.  The  mean  of 
this  distribution  is  approximately  5.2  and  the  variance  is  approximately  0.03, 


vidual  officer,  it  means  he  or  she  may 
choose  to  operate  in  accordance  with  the 
“maximin”  principle;  the  officer  must  not 
allow  any  “wart”  to  appear  in  his  or  her 
performance  record.  Thus,  a  battalion 
commander  may  feel  it  necessary  to 
spend  14“hour  days  in  the  field,  7  days  a 
week,  to  make  sure  there  are  no  “prob¬ 
lems.”  Such  an  officer  commanding  a 
unit  at  the  NTC  may  be  reluctant  to 
“innovate”  or  “experiment;”  rather,  such 
commanders  may  tend  to  “play  it  by  the 
book,”  because  it  is  risky  for  them  to 
deviate  from  doctrine.  (Being  whipped  by 
the  OPFOR  is  certainly  not  the  fault  of  a 
commander  rigorously  following  doc¬ 
trine!) 

Implications  the  elimination  process 
has  for  the  officer  corps  are  interesting. 
First,  there  is  a  perception  that  the  sur¬ 
vivors  may  tend  to  be  “conservative,”  and 
“non-risk  takers.”  Second,  the  sequence 
of  eliminations  the  survivors  must  endure 
as  they  undergo  boards  and  other 
“required”  assignments  contribute  to 
compounding  the  effects.  This  leads  to 
an  officer  corps  in  which  the  individuals 
are  nearly  indistinguishable  especially  in 
the  upper  ranks,  as  we  quantify  in  the  fol¬ 
lowing  section. 

Compounding  Elimination 
Effects:  Preliminaries 

Let  us  consider  a  single  dimension  of 
performance,  and  investigate  the  effects 
of  eliminating  candidates  failing  to 
achieve  “good  enough”  performance.  We 
want  to  examine  the  population  that 
remains  after  such  elimination,  and  to 
consider  the  effects  of  repeated  elimina¬ 
tions  by  a  sequence  of  “selection”  boards 
for  the  successively  higher  ranks.  Let  us 
first  enjoy  some  preliminaiy  ideas  based 
on  simple  calculations  with  normal  distri¬ 
butions,  assuming,  as  a  crude  model,  that 
the  original  population  of  second  lieu¬ 
tenants  has  normally  distributed  perfor¬ 
mance  scores.  For  illustration  purposes, 
we  can  assume  the  normal  distribution 
has  mean  0  and  variance  1.  The  elimina¬ 
tion  process  leads  to  a  surviving  popula¬ 
tion  with  a  truncated  normal  distribution; 
all  individuals  with  scores  below  a  given 
value  t  have  been  eliminated. 

Let  Z  be  a  standard  normal  random 
variable  truncated  below  at  a  fixed  point, 
t.  That  is,  one  could  generate  a  sample 
value  of  Z  by  sampling  from  a  N  (0,1) 


population  until  a  value  greater  than  t  is 
obtained;  that  value  is  retained  as  the  out¬ 
come  z.  Then  the  density  of  Z 
is  f{z)  =  c{t)e~^  and,  of  course, 

ffz)  is  zero  for  z  <  t;  here 
where  O  is  the 
standard  normal  cumulative  distribution 
function  (CDF).  This  density  does  not 
have  the  shape  we  might  expect,  especial¬ 
ly  for  large  truncation  points.  For  large  t, 
the  probability  density  is  quite  high  just  to 
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the  right  of  t,  and  it  decreases  veiy  rapidly 
toward  zero  as  its  argument  increases.  A 
plot  of  a  truncated  normal  density,  for  t  = 
5,  is  shown  in  Figure  1.  Thus  for  large  t 
we  would  expect  the  mean  of  Z  to  be  just 
above  t  and  the  variance  of  Z  to  be  near 
zero.  The  mean  and  variance  of  Z  are 

and  V(Z)  =  c(r) 


where  the  “+”  is  used  when  t  <  0  and 
the  “-”  is  used  otherwise.  In  the  latter 
expression,  C3  is,  curiously,  the  chi- 
square  CDF  with  three  degrees  of  free- 
dom.One  can  easily  see  that  when  the 
truncation  point  t  is  small,  say  on  the 
order  of  -5  or  smaller,  E(Z)  differs  little 
from  0,  the  mean  of  the  original  popula¬ 
tion.  As  t  increases,  E(Z)  increases,  and 
for  increasingly  large  truncation  points, 
E(Z)  approaches  t  from  above  and  V(Z) 
approaches  zero.  These  observations  are 
in  accord  with  intuition. 

Figure  2  shows  plots  of  E(Z)  and  V(Z) 
as  functions  of  the  truncation  point,  t,  and 
proportions  truncated,  ((t).  Note  the  rapid 
decrease  in  variance  as  t  increases. 

Implications  of  Elimination 
for  Higher  Ranks 

The  Army  uses  centralized  Army¬ 
wide  “selection”  boards  to  determine 
officers  qualified  for  promotion  and 
advanced  militaiy  schooling.  Each  selec¬ 
tion  board  determines  a  performance- 
based  “order  of  merit”  ranking  of  the 
officers  under  its  consideration.  As  an 
over-simplification,  officers  under  con¬ 
sideration  for  promotion  or  advanced 
schooling  that  are  not  selected  for  the 
new  grades  or  schools  either  leave  or  are 
separated  from  the  Army.  For  example, 
officers  being  considered  for  promotion 
to  Colonel  (COL,  or  “06”)  have  success¬ 
fully  passed  five  such  selection  boards. 
Data  on  success  rates  from  the  boards  in 
recent  years  (for  “primary  zone”  selec- 

(See  NON-SELECTION,/?.  22) 
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NON-SELECTION 

(continued from  p,  21) 

tion)  are  approximately  as  shown  in 
Table  1.  If  we  consider  the  population  of 
officers  considered  by  the  06  board,  we 
see  it  has  undergone  a  sequence  of  five 
truncations.  The  overall  effect  is  trunca¬ 
tion  such  that  the  fraction  (.98  x  .98  x  .74 
X  .46  X  .60)  =  0.20  of  the  original  popula¬ 
tion  is  available  for  consideration  by  the 
board. 


Table  1.  Average  primary  zone  success 
rates  for  Army  officers  in  recent  years. 


Board  Primary  zone  success  rate 

01  to  02 

.98 

02  to  03 

.98 

03  to  04 

.74 

CGSC 

.46 

04  to  05 

.60 

(There  are  losses  of  officers  for  rea¬ 
sons  other  than  non-selection,  but  non¬ 
selection,  or  the  threat  of  non-selection, 
accounts  for  most  of  the  decreases  in  the 
numbers  of  officers  at  the  succeeding 


higher  ranks.)  If  we  assume  the  original 
population  of  officers  has  a  single  dimen¬ 
sion  of  normally  distributed  performance, 
and  selection  boards  eliminate  officers 
not  having  sufficiently  high  performance, 
then  a  COL  selection  board  is  effectively 
considering  a  truncated  normal  popula¬ 
tion  of  performance,  with  truncation 
point  corresponding  to  the  80^  percentile 
of  the  original  normal  population. 

For  a  standard  normal  distribution, 
truncation  at  the  80th  percentile  would 
correspond  to  a  truncation  point,  t,  of  0. 
84.  The  variance  of  such  a  truncated  nor¬ 
mal  is  about  0.2  (see  Figure  2).  We  con¬ 
clude  the  variance  of  the  population 
under  consideration  by  the  COL  board  is 
only  about  one-fifth  that  of  the  original 
population.  This  relatively  smaller  vari¬ 
ance  makes  it  more  difficult  for  the  board 
to  discriminate  among  the  officers  under 
consideration.  Members  of  selection 
boards  for  the  higher  ranks  are  sometimes 
quoted  as  saying,  “All  the  officers  look 
about  the  same,”  and  criticisms  of  the 
officer  evaluation  report  system,  upon 
which  performance  ranks  are  based,  are 
frequently  offered.  But,  the  rapid 
decrease  in  variance  as  t  increases  goes  a 
long  way  toward  explaining  this  phenom¬ 
enon.  Even  if  the  officer  evaluation 
report  system  were  perfect,  and  used  opti¬ 
mally,  there  would  inevitably  be  relative¬ 
ly  little  difference  in  performance  scores 
at  the  higher  ranks,  due  to  the  effects  of 
truncation. 


upon  which  performance  ranks 
are  based,  are  frequently  offered. 


Conclusions 

Many  so-called  “selection”  committees 
are  actually  operating  more  like  elimina¬ 
tion  processes.  This  tends  to  produce  a  set 
of  surviving  members  that  are  middle  of 
the  road,  with  low  incentives  to  be  innova¬ 
tive  or  risk-taking.  The  compound  effects 
of  repeated  selection  boards  leads  to  sur¬ 
vivors  that  look  alike,  due  to  the  severe 
truncation  of  the  original  population.  This 
effect  will  be  present,  even  when  a  perfect 
officer  evaluation  report  system  is  used. 

If  we  are  generating  an  officer  corps 
whose  members  tend  to  strictly  follow 
doctrine,  we  must  develop  doctrine  very 
carefully.  Paradoxically,  this  must  be 
accomplished  in  the  context  of  the  “revo¬ 
lution  in  military  affairs,”  in  which  we 
encourage  bold  actions  based  on  rapidly 
developing  information  and  more  com¬ 
plete  situational  awareness. 

Notes 

a.  Actually,  the  process  is  more  compli¬ 
cated,  of  course.  A  slightly  better 
approximation  might  be  that  the  “fast 
track”  officers  are  selected  while  the 
remaining  officers  are  subjected  to 
elimination. 
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Figure  2.  Plot  of  the  mean  and  variance  of  a  truncated  standard  normal  distribution,  as  a 
function  of  the  truncation  point,  t,  and  the  fraction  truncated,  ((t). 
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The  1998  Army  Technology  Seminar  Game 


Figure  1.  Gaming  in  a  Seminar  Group 


This  is  a  report 
on  the  1998 
Army  Tech¬ 
nology  Seminar 
Game  held  27-31 
July  at  the  Center  for 
Strategic  Leadership 
(Collins  Hall)  of  the 
Army  War  College 
at  Carlisle  Barracks, 
PA.  The  purpose  of 
the  game  was  to 
determine  what  sys¬ 
tems  concepts,  and 
associated  underly¬ 
ing  technologies,  are  needed  to  support 
Army  After  Next  (AAN)  objectives. 

The  1998  Army  Technology  Seminar 
Game  (which  is  also  the  first  such  game) 
is  the  brainchild  of  LTG  Paul  Kern,  the 
Military  Deputy  to  the  Assistant  Secretary 
of  the  Army  for  Research,  Development, 
and  Acquisition,  and  is  managed  by  Dr. 
Herbert  K.  Fallin,  the  Director  of 
Assessment  and  Evaluation  in  the  office 
of  the  assistant  secretary.  The  Strategic 
Assessment  Center  of  SAIC  provided 
execution  support  under  Dr.  James 
Blackwell.  The  Game  Director  was  Mr. 
Mike  Lancaster  of  SAIC. 

The  site  host  was  MG  Robert  Scales, 
the  Commandant  of  the  US  Army  War 
College,  whose  previous  assignment  as 
TRADOC  Deputy  Chief  of  Staff  for  Doc¬ 
trine  helped  lead  the  early  development  of 
the  Army  After  Next  project.  Collins  Hall 
is  a  magnificent  war  gaming  facility  built 
since  I  was  a  student  at  the  war  college 
(Class  of  ’91.)  It  is  an  impressive  building 
of  almost  2  X  10^  ft^  and  some  large 
number  of  networked  computers.  The 
Center  for  Strategic  Leadership  does 
triple  duty  of  conducting  training  and 
investigation  war  games  and  hosting 
senior  leadership  meetings. 

We  tend  to  think  of  war  games  as  sim¬ 
ulations  with  human  beings  as  decision 
making  components.  This  game  was  a  bit 
different  (emphasis  on  seminar  here)  in 
that  the  humans  engaged  in  the  game 
were  not  coupled  directly  to  a  simulation. 
A  more  descriptive  term  might  be  “team 
dynamic  enabled  analytical  exercise”  but 
that’s  a  really  clumsy  name  so  seminar 
game  is  easier  and  convenient.  In  this 


sense,  the  players  were  not  the  usual  mix 
of  tacticians,  strategists  and  grand  strate¬ 
gists  supported  by  tactical  advisors. 

To  describe  the  structure  of  the  game, 
we  first  need  to  consider  the  nature  of  the 
AAN  project.  This  project,  chartered  by 
the  Chief  of  Staff  of  the  Army,  GEN 
Dennis  Reimer,  has  the  ambitious  pur¬ 
pose  of  elaborating  the  structure  and  form 
of  the  Army  in  2025.  The  AAN  Task 
Force,  located  at  TRADOC  Headquarters, 
Ft.  Monroe,  VA,  has  been  developing  a 
working  framework  of  the  organization, 
equipment,  tactics  and  doctrine  of  that 
future  Army.  This  work  has  been  on¬ 
going  for  about  two  years  and  has  been 
supported  by,  among  other  analytical 
efforts,  periodic  strategic  and  tactical  war 
games. 

This  in-process  product  has  reached  a 
level  of  maturity  that  the  time  had  come 
to  initiate  more  elaborate  and  definitive 
analyses  of  the  AAN  equipment  (sys¬ 
tems)  and  the  technologies  that  enable 
those  systems  by  the  technical  and  analyt¬ 
ical  communities  at  large.  This  game  was 
particularly  concerned  with  the  Army’s 
and  DoD’s  current  and  near  term  technol¬ 
ogy  bases  (largely  represented  by  the 
Army’s  Science  and  Technology  Objec¬ 
tives  (STOs)  and  DoD’s  Defense  Tech¬ 


nology  Objectives  (DTOs))  and  the  com¬ 
mercial  technology  base. 

As  a  whole,  the  Army  Technology 
Seminar  Game  consists  of  five  parts,  two 
of  which  were  conducted  at  Carlisle.  The 
first  part  consisted  of  the  preparation  for 
the  rest  of  the  game.  The  biggest  and 
most  important  product  of  this  effort  was 
a  database  that  cross  correlated  the 
vignettes  and  results  of  the  Spring  AAN 
War  Game  (a  classical  “combat”  simula¬ 
tion  war  game  at  a  combination  strategic- 
operational-tactical  level)  the  materiel  and 
information  systems  comprising  the  AAN 
force,  especially  the  new  AAN  Battle 
Force  and  its  subordinate  units,  and  the 
Army,  DoD  and  commercial  technology 
base  components  that  potentially  enable 
the  systems.  This  database  was  constmct- 
ed  by  SAIC’s  Strategic  Assessment  Cen¬ 
ter  with  several  preliminary  reviews 
including  a  workshop  sponsored  by  the 
Association  of  the  United  States  Army, 
led  by  GEN  (Ret)  Gordon  Sullivan  and 
GEN  (Ret)  Louis  Wagner,  and  attended 
by  senior  military  technologists  from  both 
Army  and  industry  laboratories  and  cen¬ 
ters. 

The  second  part  of  the  game  consisted 
of  six  gaming  sessions  for  each  of  four 
(See  SEMINAR  GAME,p.  24) 
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multidisciplinary  seminar  teams  (seminar 
in  this  case  refers  to  the  conduct  of  opera¬ 
tion  in  the  teams).  The  teams  were  com¬ 
prised  of  scientists,  military  and  commer¬ 
cial  technologists,  warfighters  and  threat 
experts.  Each  team  evaluated  the  systems 
closely  associated  with  six  vignettes 
(approximately  10  systems  per  vignette) 
by  reviewing  the  system  associated  tech¬ 
nologies  for  relevancy  and  maturity, 
adding  or  subtracting  technologies,  and  in 
some  cases,  defining  new  candidate  sys¬ 
tems.  Two  of  the  seminar  teams  were 
enjoined  to  be  technically  conservative  in 
their  review  while  the  other  two  teams 
were  to  consider  higher  risk  technologies. 
Thus  each  of  the  total  of  twelve  vignettes 
was  reviewed  twice.  The  reviews  had  two 
component  aspects:  one  a  consensual 
team  review  assessment  (with  majority 
and  minority  sub-aspects  as  necessary,) 
and  assessments  by  each  of  the  individu¬ 
als  comprising  the  team.  These  assess¬ 
ments  were  added  to  the  database  in  both 
quantitative  (scale)  and  commentary  for¬ 
mat. 

The  third  part  of  the  game  consisted  of 
a  “Hot  Wash,”  a  first  cut  review  of  the 
major  team  findings  by  a  Senior  Review 
Group  chaired  by  Dr.  William  Perry  and 
comprised  of  senior  “grey  beards”  includ¬ 
ing  Drs.  Paul  Kaminski,  and  Joe  Brad- 
dock,  Larry  Lynn  and  Paul  Decker, 
GEN  Paul  Gorman,  and  LTG  John 
Abrams,  among  others. 

These  two  parts  of  the  game  were  con¬ 
ducted  at  Carlisle  during  the  week  of  27- 
31  July  1998.  The  fourth  and  fifth  parts  of 
the  game  will  consist  of  detailed  analyses 
of  the  data  base  developed  during  the  first 
through  third  parts  of  the  game.  The 
fourth  part,  to  be  performed  by  SAIC, 
will  consist  of  a  technology  oriented  sta¬ 
tistical  and  outlier  analysis  of  the  data 
base.  (The  outlier  analysis  is  intended  to 
specifically  address  potential  technologi¬ 
cal,  organizational,  or  military  “gotchas.”) 
The  fifth  part,  to  be  performed  by  RAND, 
will  be  a  system-of-systems  oriented 
analysis  of  the  database. 

For  the  analytical  and  decision  support 
community,  the  1998  Army  Technology 
Seminar  Game  signals  a  change  in  the 
scope  and  optempo  of  efforts,  the  first  in 
a  series  of  efforts  for  the  future.  With  the 
relative  maturity  of  AAN,  consideration 


of  its  structure  now  naturally  extends 
beyond  the  AAN  task  force  to  increasing¬ 
ly  include  not  only  the  research,  develop¬ 
ment,  and  acquisition,  logistics,  training 
and  manning  organizations  of  the  Army 
but  of  the  other  services  as  well.  The  lat¬ 
ter  follow  from  the  implicitly  and  inher- 


I  The  1998  Technology  Seminar  Game 
'  presents  the  analytical  community 
with  considerable  opportunities. 
Determining  what  systems  will 
equip  the  future  Army  is 
more  than  business  as  usual. 


ently  joint  form  of  AAN  operations;  the 
former  follows  from  the  need  to  support 
this  future  Army  with  materiel,  training, 
doctrine  and  sustainment.  Further  analyses 
will  build  on  this  initial  effort  to  support 


this  development.  The  results  of  these 
analyses  will  be  used  to  plan  and  imple¬ 
ment  the  research,  development  and 
acquisition  of  future  systems  for  the 
Army. 

The  nature  of  these  analyses  seem 
likely  to  be  more  joint  and  combined  than 
past  analyses  because  of  the  tight  cou¬ 
pling  not  only  among  Army  processes, 
but  with  other  service  processes  as  well. 
In  the  past,  deployment  and  sustainment 
analyses  have  often  been  decoupled  from 
combat  analyses,  and  combat  analyses 
have  decoupled  force  structure  and  mater¬ 
ial  assessments. 

The  1998  Technology  Seminar  Game 
presents  the  analytical  community  with 
considerable  opportunities.  Determining 
what  systems  will  equip  the  future  Army 
is  more  than  business  as  usual.  It  will 
require  more  comprehensive  analyses  of 
these  systems  and  their  enabling  tech¬ 
nologies  and  closer  interaction  between 
the  engineering  and  operational  analysis 
communities  in  their  analysis  efforts. 
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1998  Vance  R.  Wanner  Memorial  Award 
Presented  to  General  Larry  D.  Welch 


March  1998  Tisdale 
Award  Recipient 


Citation 

Military  operations  research  contributes 
effectively  to  the  nation’s  security  to  the 
degree  that  it  meets  the  evolving  needs  of 
defense  and  conforms  to  the  highest  stan¬ 
dards  of  scientific  quality  and  timeliness. 
To  grant  recognition  to  individual  practi¬ 
tioners  who  have  made  outstanding  contri¬ 
butions  to  the  progress  of  this  advancing 
profession,  The  Military  Operations 
Research  Society  has  estabhshed  the  Vance 
R.  Wanner  Memorial  Award,  The  Directors 
of  the  MORS  have  selected,  for  this  award 
in  1998: 

•  Larry  D.  Welch 

Who  has  applied  a  rare  combination  of 
vision,  leadership  and  analytical  expertise 
to  our  most  urgent  and  persistent  national 
security  problems.  The  principles  of  mili¬ 
tary  analysis  and  the  associated  ethics  has 
been  an  integral  part  of  his  leadership  roles 
at  the  highest  level  of  each  of  the  organiza¬ 
tions  he  has  served. 

His  career  in  the  Air  Force  started  in 
1953  and  culminated  in  1986  when  he 
became  Chief  of  Staff  of  the  Air  Force,  It 
continues  with  his  visionary  leadership  of 
the  Institute  for  Defense  Analyses,  together 
with  his  broad-ranging  contributions  to  the 
national  security  community  overall.  In  the 
late  1960’s  he  developed  the  Air  Force’s 
first  air-to-air  combat  model  and  used  it  to 
help  define  the  F-15  program.  His  support 
of  the  Military  Operations  Research  Soci¬ 
ety  includes  articles  of  effective  analysis 
and  presenting  the  keynote  at  a  Sympo¬ 
sium. 

His  understanding  of  military  operations 
has  led  others  to  seek  his  counsel,  as  is  evi¬ 
denced  by  his  leadership  and  membership 
on  numerous  appointed  national-level  advi¬ 
sory  boards  and  commissions. 

His  interests  and  influence  have  extend¬ 
ed  to  the  highest  levels  of  our  government, 
always  marked  by  an  objective  and  analyti¬ 
cally  sound  basis.  For  his  entire  multifac¬ 
eted  career  he  has  been  a  significant  force 
in  advising  and  shaping  the  national  securi- 


Dr.  William  Perry  presents  General 
Larry  Welch  (Ret)  with  the  Vance  R. 
Wanner  Memorial  Award. 


ty  agenda,  both  from  a  military  and  a  civil¬ 
ian  perspective. 

General  Welch’s  dedication  to  country, 
visionary  leadership,  personal  integrity  and 
dedication  to  quality  analysis  exemplifies 
the  spirit  and  ideals  of  the  Vance  R.  Wan¬ 
ner  Memorial  Award. 

General  Welch’s 
Acceptance  Remarks 

Thank  you.  I  deeply  appreciate  the 
honor  and  thank  you  without  reservation. 
Still,  I  feel  I  have  to  tell  you  that  there  is  a 
certain  irony  associated  with  this  occasion. 
When  I  finished  a  combat  tour  while  in  the 
Air  Force  in  Vietnam  and  received  orders 
from  the  Air  Force  Personnel  System  that  I 
was  to  report  to  the  Assistant  Chief  of  Staff 
for  Studies  and  Analyses,  I  regarded  that 
decision  as  a  confirmation  of  what  a  lot  of 
fighter  pilots  have  always  suspected  —  that 
the  Air  Force  Personnel  System  was  served 
by  a  decision-aid  that  consisted  of  a  dart 
board  and  darts  provided  by  the  lowest  bid¬ 
der.  It  was  beyond  the  bounds  of  my  imag¬ 
ination  that  I  could  belong  to  anything 
called  studies  and  analyses.  My  first  meet¬ 
ing  with  General  Glenn  Kent  verified  that 
judgment.  The  irony,  of  course,  is  that  I 
spent  five  years  in  this  three-year  assign- 

(See  WELCH,  p.  26) 


OPTIMIZING  THE  SELECTION  OF 
TOMAHAWK  CRUISE  MISSILES 
Scott  D.  Kuykendall-Lieutenant, 
United  States  Navy 

B.S.,  United  States  Naval  Academy,  1992 
Master  of  Science  in  Operations 
Research-March,  1998 
Advisor  Richard  E.  Rosenthal, 
Department  of  Operations  Research 

The  Tomahawk  Land  Attack 
Cruise  Missile  (TLAM),  launched 
from  surface  ships  and  sub¬ 
marines,  has  become  the  weapon  of 
choice  for  the  United  States  in  many  situ¬ 
ations.  In  an  era  of  high  precision,  fast 
delivery  weapons,  the  method  currently 
used  for  assigning  TLAM  engagements 
is  out  of  step  with  the  development  of  the 
weapons  themselves.  The  missile  assign¬ 
ment  process  used  today  is  essentially 
manual,  with  the  potential  consequences 
of  inefficient  missile-to-mission  matching 
the  critical  loss  of  time. 

This  thesis  develops  a  new  optimizing 
approach  to  missile-to-mission  matching, 
using  integer  programming.  In  a  matter  of 
seconds  for  a  single  ship  or  a  matter  of 
minutes  for  a  battle  group,  the  optimiza¬ 
tion  model  determines  which  missile  to 
select  for  each  tasking  order  and  provides 
back-up  assignments  if  requested.  The 
objective  of  the  model  is  to  ensure  the 
best  weapon  is  applied  against  each  target 
while  maximizing  the  potential  of  the  fir¬ 
ing  unit(s)  to  perform  future  taskings. 

The  new  missile-to-mission  matching 
model  has  proven  to  perform  robustly  in 
extensive  sensitivity  analysis  and  is  cur¬ 
rently  being  considered  for  shipboard 
implementation  by  the  Naval  Surface 
Warfare  Center. 

KEYWORDS:  Tomahawk  Land  Attack 
Cruise  Missiles  (TLAM),  Vertical 
Launch  System  (VLS),  Missile  Selection, 
Missile-to-Mission  Matching  (M3) 
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ment  and  I  now  regard  it  as  among  the  most 
important  and  productive  of  my  profession¬ 
al  life.  In  subsequent  assignments  I  never 
escaped  from  the  world  of  Systems  Analy¬ 
sis  and  Operations  Research,  nor  did  I  ever 
try  to  do  so.  Among  the  many  labels  that 
have  been  attached  to  me  over  the  years, 
some  of  which  I  can’t  relay  in  polite  com¬ 
pany,  one  that  I  value  greatly  is  “Defense 
Analyst.” 

At  the  time  that  I  joined  the  Fighter 
Division  of  Air  Force  Studies  and  Analy¬ 
ses,  we  were  entering  a  new  era  with  the 
lessons  of  Vietnam,  the  emerging  technolo¬ 
gies,  many  of  which  Dr.  Perry  mentioned, 
and  the  cost  and  potential  effectiveness  of 
new  capabilities  that  were  on  the  horizon. 
These  led  to  a  new  set  of  demands  on  the 
world  of  operations  research  that  I  think 
was  unprecedented,  at  least  certainly 
unprecedented  since  the  end  of  World  War 
n.  And  I  suggest  to  everyone  here  that  the 
contributions  the  world  of  military  opera¬ 
tions  research  made  to  the  decision  process¬ 
es  were  very  strongly  reflected  in  the  per¬ 
formance  and  the  capabilities  of  the  forces 
that  we  saw  in  Desert  Storm. 

We  are  now  in  an  era  where  there  is 
even  a  greater  demand  for  objective  analy¬ 
sis  that  will  help  us  understand  the  implica¬ 
tions  of  new  technologies  and  truly  revolu¬ 
tionary  new  capabilities,  to  include  the 
information  technologies  that,  again.  Dr. 
Perry  explained  so  well.  These  demands 
make  the  analysis  needs  of  earlier  periods 
pale  in  comparison.  I  know  of  no  time 
when  defense  decision  makers  have  been 
faced  with  a  more  difficult  set  of  choices, 
and  a  budget  that  requires  these  choices. 
The  capabilities  that  are  reflected  in  the  sys¬ 
tems  they  must  choose  have  the  potential 
for  enormous  impact  on  national  security 
and  on  the  capabilities  of  our  national 
defense  apparatus.  This  adds  up  to  an 
unprecedented  need  for  objective,  innova¬ 
tive,  disciplined  analysis  to  help  make  the 
decisions  that  will  convert  Joint  Vision 
2010  into  Joint  Capability  2010  and 
beyond.  It  is  that  challenge  that  keeps  me 
in  Defense  Analysis  long  after  I  expected  to 
have  left  the  business.  It  keeps  me  in  the 
business  and  it  gives  me  a  very  special 


MAS  Award  to  MIDN  Detar 

Midshipman  First  Class  Paul  J. 

Detar  received  the  1998  award  spon¬ 
sored  by  the  Military  Application  Soci¬ 
ety  (MAS)  as  the  Naval  Academy’s  out¬ 
standing  midshipman  in  operations 
research.  The  award  certificate  and  a 
check  for  $300  were  presented  to  Mid¬ 
shipman  Detar  in  a  ceremony  during 
graduation  week  on  19  May  1998  at  the 
Naval  Academy. 

The  award  was  presented  by  Dr. 

Thomas  Frazier  of  the  Institute  for 
Defense  Analyses,  on  behalf  of  MAS. 

As  the  basis  for  this  award.  Dr.  Frazier 
cited  Midshipman  Detar’s  outstanding 
record  in  operations  analysis  courses 
capped  by  an  excellent  senior  project. 

His  project  was  entitled  “Save  Money 
for  the  Midshipmen  Wardroom  Mess  by 
Forecasting.”  His  advisor  was  Associate 


(From  left  to  right)  Dr.  William  Miller,  Provost  and  Academic  Dean;  Captain 
Howard  J.  Halliday,  Director  of  the  Division  of  Mathematics  and  Science;  Mid¬ 
shipman  Paul  J.  Detar;  Dr.  Thomas  Frazier  of  The  Institute  for  Defense  Analyses 
representing  the  Military  Applications  Society. 


Professor  Gary  Fowler. 

In  addition  to  his  MAS  award,  Mid¬ 
shipman  Detar  was  given  the  Military 
Order  of  Foreign  Wars  Prize  for  the 
mathematics  major  achieving  the  highest 
grade  point  average  in  mathematic 
courses. 

Midshipman  Detar  graduated  on  22 
May  with  a  major  in  Mathematics  and 
was  commissioned  a  Second  Lieutenant 
in  the  US  Marine  Corps.  He  has  been 
awarded  a  Burke  Scholarship  and  will  be 
allowed  to  pursue  a  Masters  Degree  after 
completing  his  first  operational  tour.  He 
is  considering  the  Operations  Research 
program  at  the  Naval  Postgraduate 
School. 

Midshipman  Detar  comes  from 
Greensburg  PA. 


appreciation  for  any  association  with  an 
organization  that  dedicates  itself  to  provid¬ 
ing  the  kind  of  decision  support  that  helps 
ensure  national  security  for  ourselves,  our 


children  and  our  grandchildren. 

I  thank  you  for  this  award  and,  more 
importantly,  I  thank  you  for  your  part  in  our 
national  security.  O 
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A  New  Name ...  A  New  Home 


E.  B.  Vandiver  in 

Director,  Center  for  Army  Analysis 

n  1  October  1998  the  US  Army 
Concepts  Analysis  Agency 
(CAA)  became  the  Center  for 
Army  Analysis.  Still  CAA,  but 
with  a  new  meaning.  On  25 
March  1999  the  new  CAA 
will  move  to  a  new  home 
in  a  newly  constructed 
building  at  Fort  Belvoir, 

Virginia. 

Two  years  ago  the 
Headquarters  Redesign 
Study  Group  made 
changes  to  the  size,  func¬ 
tions,  and  organization  of 
the  Headquarters  of  the 
Department  of  the  Army  to  realign 
it  with  the  downsizing  that  had  already 
occurred  in  the  field.  For  CAA  this 
entailed  assuming  the  logistics  analysis 
functions  of  the  former  Logistics  Evalua¬ 
tion  Agency,  a  slight  adjustment  in  man¬ 
power  and  the  change  of  name  to  the  Cen¬ 
ter  for  Army  Analysis. 

The  Base  Realignment  and  Closure 
Commission  of  1995  found  that  it  would 
be  economical  to  move  CAA  from  com¬ 
mercial  leased  space  in  Bethesda,  Mary¬ 
land  to  either  renovated  or  new  construc¬ 
tion  at  Fort  Belvoir,  Virginia.  Since  all 


existing  structures  capable  of  renovation 
had  already  been  designated  for  other  orga¬ 
nizations,  new  construction  was  undertak¬ 
en.  Ground  was  broken  on  3 
November  1997  with 
building  completion  by 
the  end  of  1998.  After 
a  period  of  fit  up  of 
communications  and 
furnishings,  occupan¬ 
cy  is  set  for  the  plan¬ 
ning  date  of  25  March 
1999.  The  accompa¬ 
nying  photograph 
shows  the  new  home  of 
CAA  as  it  appeared  on  3 
October  1998. 

The  new  CAA  building  will  be  named 
Wilbur  B.  Payne  Hall  memorializing  the 
late  Dr.  Wilbur  B.  Payne,  first  Deputy 
Under  Secretary  of  the  Army  (Operations 
Research)  and  first  Director  of  the 
TRADOC  Systems  Analysis  Agency 
(TRASANA).  Payne  Hall  will  be  dedicat¬ 
ed  on  21  May  1999  with  the  Payne  family 
in  attendance.  The  dedication  ceremony 
will  include  the  unveiling  of  an  artist’s  por¬ 
trait  of  Dr.  Payne  that  has  been  graciously 
commissioned  by  the  TRADOC  Analysis 
Center,  today’s  successor  organization  to 


TRASANA, 

In  little  over  a  one  year  period  CAA 
will  have  celebrated  its  Silver  Anniversary 
as  the  US  Army  Concepts  Analysis 
Agency;  changed  its  name  to  the  Center  for 
Army  Analysis;  moved  to  a  new  home  in  a 
new  building;  and  memorialized  Dr. 
Wilbur  Payne,  one  of  the  giants  of  our  pro¬ 
fession.  O 


TISDALE 
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DOD  KEY  TECHNOLOGY  AREA: 

Computing  and  Software,  Conventional 
Weapons 

Biography 

Lieutenant  Scott  D,  Kuykendall  gradu¬ 
ated  from  South  Carroll  High  School  in 
Sykesville,  Maryland  in  1988.  He  was 
accepted  as  an  early  applicant  to  the  Unit¬ 
ed  States  Naval  Academy  and  entered  the 
Academy  in  July  1988.  He  graduated  with 
a  Bachelor  of  Science  degree  in  Oceanog¬ 
raphy,  and  was  commissioned  as  an 
Ensign  in  the  United  States  Navy  on  27 
May  1992. 

Lieutenant  Kuykendall  selected  Surface 
Warfare  and  was  assigned  for  duty  on 
board  the  USS  O’Bannon  (DD-987)  out  of 
Charleston,  South  Carolina.  En  route,  he 
attended  Surface  Warfare  Officer  School 
Division  Officer  Course,  Legal  officer 
School,  Damage  Control  Assistant  School 
and  Tomahawk  Weapons  Officer  School. 
While  on  board  the  O'Bannon  he  was 
assigned  duties  as  Damage  Control  Assis¬ 
tant  for  24  months  and  Strike  Warfare  Offi¬ 
cer  for  12  months.  He  did  one  deployment 
to  the  Persian  Gulf  and  one  Counter-drug 
Operation  in  the  Caribbean  and  Eastern 
Pacific  Ocean. 

Following  his  duties  on  board  the 
O'Bannon,  he  reported  to  the  Naval  Post¬ 
graduate  School  in  February  1996. 

His  next  duty  assignment  is  at 
SWOSCOLCOM,  Newport,  Rhode  Island, 
and  subsequently  as  Combat  Systems  Offi¬ 
cer  on  board  the  USS  Spruance. 

He  is  married  to  the  former  Melissa 
Lowe  of  Mt.  Airy,  Maryland  and  has  two 
children.^ 


The  new  CAA  building. 
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current?  The  configuration  control  and 
maintenance  of  the  models  and  simulations 
are  critical:  to  validate  input  to  virtual  envi¬ 
ronments,  to  interface  with  different 
threats,  to  interact  with  other  weapon  sys¬ 
tems  and  to  modernize  the  weapon  system. 
For  example,  if  a  piece  of  kit  is  fielded  and 
a  training  simulation  developed  that  is 
based  on  the  currently  fielded  system,  what 
happens  if  the  piece  is  modernized  by 
spare  parts?  As  we  build  new  systems,  or 
modernize  by  spares,  it  will  become  more 
critical  to  build  hooks  into  the  weapon  sys¬ 
tem  that  will  enable  us  to  stimulate  the  sen¬ 
sors  or  hook  into  physics  models  as  in  a 
virtual  proving  ground. 

Another  area  of  concern  is  to  what 
extent  logisticians  can  capitalize  on  the 
simulations  by  using  them  to  discover  what 
might  break.  Historically,  reliability  engi¬ 
neers  have  predicted  system  reliability 
using  statistical  distributions  of  the  histori¬ 
cal  failure  rates.  Through  the  use  of 
physics  based  modeling  techniques,  we  can 
now  better  analyze  the  failure  mechanisms 
and  complex  root  cause  failure  modes 
associated  with  our  systems  in  the  opera¬ 
tional  environment.  One  of  the  advantages 
of  a  “Physics  of  Failure”  approach  to  relia¬ 
bility  prediction  is  that  it  can  be  applied 
earlier  in  the  design  process  to  analyze  fail¬ 
ure  frequency,  cost  of  failure  or  criticality 
of  failure  and  to  optimize  the  design  for 
reliabihty  performance. 

To  accomplish  this,  we  need  to  evolve 
physics  based,  stress  related  failure  predic¬ 
tion  models  that  can  be  used  to  identify 
failure  prone  components  and  redesign 
them  to  optimize  reliability  characteristics, 
enabling  increases  in  reliability  growth 
during  the  development  phases.  Better 
understanding  the  mechanics  of  failure  will 
allow  closer  prediction  of  when  a  failure  is 
likely  to  occur.  Knowing  this,  the  logistics 
community  benefits  by  better  spare  part 
prediction  and  performing  proactive  main¬ 
tenance  to  replace  bad  components  before 
they  fail,  avoiding  much  costlier  cata¬ 
strophic  failures. 

Some  of  the  other  challenges  that  com¬ 
puter  scientists  can  explore  are  providing 
high  fidelity,  real  world  simulations:  total 
ownership  cost  modeling;  manufacturing; 
logistics;  developing  realistic  training  sim¬ 
ulations  (including  fog  of  war)  in  virtual 
environments;  developing  dual  use  simula¬ 


tions  for  both  acquisition  and  training;  and, 
continued  development  of  representations 
of  an  entity  as  an  object  or  as  a  set  of  per¬ 
formance  tables  in  WARSIM,  an  Army 
warfighting  simulation. 

A  benefit  of  computer  simulation  not 
yet  actively  pursued  is  harnessing  the  latest 
processing  capability  on  board  weapon 
systems  for  training,  repair  diagnostics, 
condition  monitoring,  etc.  The  automobile 
industry  is  utilizing  this  capability  by  uti¬ 
lizing  on  board  diagnostics  when  we  take 
our  car  in  for  servicing. 

Transition  to  the  Future.  There  are 
many  efforts  in  the  Defense  analysis  com¬ 
munity  that  will  impact  the  manner  in 
which  we  will  do  business  in  the  future. 

Conducting  acquisition  digitally  implies 
access  to  work  in  progress  across  the  func¬ 
tional  areas  in  the  weapon  system;  training, 
requirements,  logistics,  testing,  as  well  as 
access  to  the  smart  product  models  and 
integrated  environments.  This  raises  issues 
of  proprietary  information,  configuration 
control,  M&S  ownership,  access,  simula¬ 
tion  licensing  and  maintenance  of  the  mod¬ 
els. 

The  operations  research  analyst  must 
stay  involved  in  each  of  the  acquisition 
functions;  more  robust  use  of  simulations 
will  continue  to  demand  more  analysis  in 
support  of  the  weapon  system  program. 
We  in  the  analysis  community  must  also 
grapple  with  the  impact  that  some  of  the 
super  models  will  have  on  analysis; 
JWARS,  JSMS  and  WARSIM.  Business 
in  the  future  will  be  based  on  DoD  integra¬ 
tion  of  M&S  into  all  functional  areas  of 
analysis,  and  the  creation  of  a  seamless 
interface  between  DoD  and  Industry. 
Acquisition  reform  is  guiding  each  Service 
as  it  moves  forward,  tearing  down  the  bar¬ 
riers  between  conunercial  Industry’s  suc¬ 
cesses  and  the  way  we  used  to  procure 
platforms  for  the  warfighters.  The  technol¬ 
ogy  of  the  future  demands  that  the  Army 
and  DoD  change  the  way  they  do  business 
to  maximize  benefits  and  stay  abreast  of 
the  best  practices  available.  The  analysis 
community,  as  the  catalyst  to  make  M&S 
an  effective  tool,  must  find  or  create  the 
tools  which  will  support  this  dramatic 
change. 

The  tools  that  best  serve  the  analysis 
community  need  to  foster  interoperability 
and  reusability  so  that  the  benefit  is  maxi¬ 
mized  to  all  Services  and  functional  areas. 
The  creation  of  some  standards  will  be 


necessary  in  order  for  everyone,  DoD  and 
Industry  alike,  to  communicate  and  “level 
the  playing  field.”  The  Army  is  aggres¬ 
sively  pursuing  standards  in  all  areas  of 
M&S.  Likewise,  the  Joint  community  is 
concentrating  model  development  on  tools 
that  will  ensure  standards  and  interoper¬ 
ability.  These  models  include  JSIMS, 
JWARS,  and  WARSIM.  JSIMS  will  pro¬ 
vide  a  readily  available,  valid,  synthetic 
training  environment  for  CINCs,  their 
components  and  joint  organizations  to 
assess  warfighting  situations  and  define 
operational  requirements.  The  Army’s 
Warfighter’s  Simulation  2000  (WARSIM) 
is  the  flagship  training  simulation  to  be 
used  as  ground  forces  representation  in 
JSIMS.  JWARS  will  provide  a  fully  inte¬ 
grated,  state  of  the  art,  joint  warfare  model 
for  CINCs  which  will  represent  unique 
joint  functions,  processes  and  component 
warfare  operations,  and  will  aid  in  force 
structure  and  courses  of  action  analysis. 
These  two  joint  programs  are  investigating 
a  Conceptual  Models  of  the  Mission  Space 
(CMMS),  which  sets  the  stage  for  a  stan¬ 
dard  that  further  integrates  training  and 
analysis,  while  maximizing  effectiveness 
and  efficiency.  Tools  and  standards  which 
allow  seamless  communication  and  inte¬ 
gration  between  Services,  and  between 
DoD  and  Industry,  wiU  promote  a  smooth 
transition  to  SBA  and  avoid  many  of  the 
pitfalls. 

Chaise  to  the  Community.  What  can 
the  analysis  community  do  to  assist  the 
transition  to  Simulation  Based  Acquisi¬ 
tion?  Continue  to  provide  and  pursue  real¬ 
istic  ideas  on  how  to  maintain  weapon  sys¬ 
tem  simulations  under  configuration 
control.  Our  analysts  can  also  monitor 
development  of  physics  of  failure  method¬ 
ology  which  can  then  be  coupled  with 
intelligent  systems  and  prognostics  to 
accurately  predict  the  useful  operating  life 
of  components  on  a  fielded  system. 

You  also  have  the  opportunity  to  make 
a  difference  by  participating  in  the  devel¬ 
opment  of  some  of  the  standards  for  mod¬ 
els  used  in  the  Army.  Standardizing  the 
Army’s  Model  and  Simulation  (M&S) 
processes  is  a  vital  step  toward  achieving 
the  economies,  efficiencies  and  technologi¬ 
cal  potential  M&S  represents.  The  Army 
Model  and  Simulation  Office  (AMSO) 
leads  standards  groups  consisting  of  Indus¬ 
try,  Government  and  Academia  who  can 
submit  ideas  for  standards.  To  further  this 
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AFORS 1998  Sets  Attendance  Record 

Dr.  Roy  Rice 

Teledyne  Brown  Engineering 


process,  AMSO  has  developed  the  Stan¬ 
dards  Nomination  and  Approval  Process 
(SNAP),  a  web-based  tool  designed  to 
track,  promote  discussion,  and  vote  on  draft 
M&S  standards.  After  a  standard  has  been 
approved,  it  is  placed  in  the  Army  Stan¬ 
dards  Repository  System  (ASTARS).  Both 
of  these  systems  are  available  through  the 
Army  node  of  the  Model  and  Simulation 
Resource  Repository  and  the  AMSO  Home 
Page  and  I  encourage  you  to  become 
involved  in  these  processes. 

There  exists  today  a  need  for  better  tools 
and  more  robust  databases  to  support 
weapon  system  technologies.  However 
well  we  develop  these  simulation  tools  in 
support  of  Simulation  Based  Acquisition, 
there  remains  a  need  for  rigorous  analysis 
of  the  results  of  the  simulations... and  well 
trained  operations  research  personnel  to 
execute  these  analyses. 

Biography 

Mr.  Walter  W.  Hollis  graduated  in 
1949  from  Northeastern  University  in 
Boston,  Massachusetts,  with  a  Bachelor 
of  Science  degree.  Following  graduation 
Mr.  Hollis  taught  in  the  Physics  Depart¬ 
ment  at  Northeastern  and  engaged  in 
graduate  study  at  Boston  University.  In 
1951  Mr.  Hollis  entered  the  Civil  Service 
as  an  Optical  Engineer  at  Frankford 
Arsenal,  PA  where  he  held  progressively 
more  responsible  positions  for  17  years. 
In  1968  Mr.  Hollis,  then  Chief,  Combat 
Vehicle  and  General  Instruments  Fire 
Control  Laboratory,  became  Scientific 
Advisor  to  the  Commanding  General,  US 
Army  Combat  Developments  Experimen¬ 
tation  Command,  Fort  Ord,  CA,  a  posi¬ 
tion  he  held  until  1972  when  he  became  a 
student  at  the  National  War  College 
(NWC).  In  1973,  after  graduation  from 
NWC  and  receiving  a  Master  of  Science 
in  International  Affairs  from  George 
Washington  University,  Mr.  Hollis 
assumed  his  position  as  Scientific  Advisor 
to  the  Commanding  General,  US  Army 
Operational  Test  and  Evaluation  Agency, 
Falls  Church,  VA.  He  assumed  his  pre¬ 
sent  post  of  Deputy  Under  Secretary  of 
the  Army  (Operations  Research)  in 
December  1980. 

Among  the  many  awards  that  Mr.  Hol¬ 
lis  has  received  are  three  Presidential 
Meritorious  Executive  Awards,  one  Presi¬ 
dential  Distinguished  Executive  award, 
and  three  Department  of  the  Army  Excep¬ 
tional  Civilian  Service  Awards.  O 


The  Air  Force  held  its  Fifth  Annual 
Air  Force  Operations  Research 
Symposium  (AFORS  ‘98)  at  the 
United  States  Air  Force  Academy,  Octo¬ 
ber  1st  and  2nd,  1998.  The  symposium 
featured  professional  exhanges,  briefings 
on  military  and  civilian  career  manage¬ 
ment,  panel  discussions  and  updates  on 
current  AF  policies  on  analysis,  modeling 
and  simulation.  This  was  the  most 
attended  AFORS  (over  130  attendees)  to 
date  and  was  sponsored  by  the  Space  and 
Missile  Systems  Center  (SMC). 

Specific  briefings  to  the  entire  group 
included  presentations  on 

•  Global  Engagement  Operations 
(GEO) 

•  Expeditionary  Air  Force  (EAF) 

•  AF  Standard  Analysis  Toolkit 
•  JWARS:  An  Air  Force 
Perspective 

Breakout  sessions  were  held  where 
over  25  classified  and  unclassified  brief¬ 
ings  were  presented  by  individuals  repre¬ 
senting  seven  organizations. 

A  panel  discussion  with  Q&A  on  “The 
State  of  AF  Analysis”  was  led  by  Col 
Kurt  Cichowski  (AFSAA),  Mr.  Dave 
Merrill  (AMC),  Col  Jerry  Levesque 
(ACC),  Col  Richard  Gibson 

(ESC),  Col  Dan  Litwhiler  (USAFA/ 
DFMS),  Lt  Col  John  Miller  (AFIT/ 
ENS),  and  Lt  Col  Jerry  Diaz  (OAS). 

The  Keynote  Speaker  for  the  AFORS 
‘98  was  Dr.  David  Finkleman,  Director 
of  Studies  and  Analysis  and  Senior  Scien¬ 
tist,  North  American  Aerospace  Defense 
Command  and  U.S.  Space  Command, 
His  address  on  the  history  of  modeling 
space  assets  was  highhghted  by  his  per¬ 
sonal  advice  and  guidance  on  conducting 
OR  analysis  in  general. 

At  the  Awards  Banquet,  Dr.  Jackie 
Henningsen,  FS,  Associate  Director  for 
Modeling,  Simulation,  and  Analysis, 
Directorate  for  Command  and  Control, 
Deputy  Chief  of  Staff  for  Air  and  Space 
Operations,  HQ  USAF,  delivered  an 
address  on  Analysis  Lessons  Learned 
from  the  QDR.  Dr.  Henningsen  tasked 


the  Air  Force  analysis  community  to  start 
getting  ready  now  for  the  next  QDR. 

At  the  AFORS  ’98  Awards  Banquet, 
Col  Kurt  Cichowski,  Commander  of 
AFSAA,  awarded  four  Analyst  of  the 
Year  awards.  They  were  awarded  to  the 
Company  Grade  and  Field  Grade  Military 
Analysts  of  the  Year  and  the  Junior  and 
Senior  Civilian  Analysts  of  the  Year. 

The  Junior  Civilian  Analyst  of  the 
Year  was  Mrs.  Nancy  L.  Evans,  Air 
Combat  Command,  Studies  and  Analyses 
Squadron.  The  Company  Grade  Mihtary 
Analyst  of  the  Year  was  Capt  Dave  Lyle, 
Office  of  Aerospace  Studies.  The  Senior 
Civilian  Analyst  of  the  Year  was  Mr. 
Patrick  J.  McKenna,  USSTRAT 
COM/J5B.  The  Field  Grade  Analyst  of 
the  Year  was  Maj  Mark  A.  Gallagher, 
USSTRATCOM/J5B. 

Also,  AFORS  awarded  two  Lifetime 
Achievement  Awards.  One  award  went 
to  Dr.  Tom  Allen  (Col  (ret.)),  who 
recently  retired  as  the  Commander  of 
AFSAA.  The  other  award  went  to  Mr. 
Clayton  J.  Thomas,  FS,  Chief  Scientist 
of  AFSAA,  for  over  56  years  of  dedicated 
service  in  Operations  Research.  O 


VEEPS  PEEP 

(continued from  p.  5) 

ing  process. 

We  have  a  wealth  of  experience  in 
addressing  problems  and  we  need  to 
help  decision  makers  understand  we  are 
limited  only  by  the  decision  makers 
willingness  to  include  us  in  the  process. 
By  breaking  down  this  barrier  and 
showing  our  capabilities  to  address  the 
full  range  of  problems  —  we  will  finally 
begin  to  achieve  the  true  potential  of  our 
profession. 
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HONORS 


1998  Air  Force  Operations  Research  Analyst  of  the  Year  Awards 

Maj  Harry  Newton  (US  AF  Academy) 

Maj  Robert  Morris  (Air  Force  Studies  and  Analyses  Agency) 


The  Analyst  of  the  Year  Awards  are 
given  out  annually  by  the  Air  Force 
Analytical  Community,  chaired  by 
Col  Kurt  A.  Cichowski,  Commander  of  the 
Air  Force  Studies  and  Analyses  Agency. 
Air  Force  units  are  allowed  to  nominate  one 
analyst  in  each  category.  This  year  a  total  of 
28  analysts  were  nominated.  Selection  of 
the  award  winners  was  made  by  a  panel  of 
peers  with  oversight  by  AFSAA.  The 
awards  were  presented  at  the  Analyst  of  the 
Year  Dinner  during  AFORS  (Air  Force 
Operations  Research  Symposium)  held  at 
USAF  Academy,  1-2  October  1998.  Over 
120  analysts  attended  the  dinner. 

Winners  of  the  1998  Air  Force  Analysts 
of  the  Year  Award: 

Field  Grade:  Maj  Mark  A,  Gallagher 
(USSTRATCOM) 

Company  Grade:  Captain  Dave  Lyle  (OAS) 
Senior  Civilian:  Mr  Patrick  McKenna 
(USSTRATCOM/J5B) 

Junior  Civilian:  Mrs  Nancy  Evans  (HQ 
ACC/XP-SAS) 

Other  nominees  were: 

Field  Grade:  Maj  William  K.  Adams  (HQ 
ACC/XP-SAS),  Maj  Paul  S.  Copeland 
(AFSAA/CC),  Maj  David  A.  Fulk 
(AFLMA/CC),  Maj  Whitney  J.  Hulett 
(SMC/XR),  Lt  Col  Lee  J.  Lehmkuhl 
(HQ  USAFA/DFMS),  Maj  Mark  L. 
MacDonald  (HQ  USAF/  XPY),  Lt  Col 
Mark  D.  Reid  (HQ  AFOTEC/CC),  Maj 
Richard  W.  Roberts  (OAS) 

Company  Grade:  Capt  Robert  M.  Block 
(USAFA),  Capt  James  R.  Hunter 
(SMC/XR),  Capt  Brian  E.  Ralston  (HQ 
AFOTEC/CC),  2Lt  Michael  A.  Rosen¬ 
baum  (ESC/DIS),  ILt  Joshua  D.  Snod¬ 
grass  (Det  1,  31  TES),  Capt  Thomas  J. 
Timmerman  (AFSAA/CC),  Capt 
Michael  J.  Wall  (HQ  ACC/XP-SAS) 

Senior  Civilian:  Ms.  Nancy  L.  Clements 
(ASC/XR),  Mr.  Stephen  R.  Ganger  (HQ 
AFOTEC/CC),  Mr.  David  M.  Hickman 
(HQ  ACC/XP-SAS),  Mr.  Keenen  Kloep- 
pel  (OAS),  Dr.  James  K.  Lowe  (HQ 


Dr.  Jacqueline  Henningsen,  FS  (right)  and  Col  Cichowski  (left)  hosted  the  Analyst  of 
the  Year  Dinner  held  to  honor  the  Nominees  for  this  Award.  From  left  to  right  Col 
Cichowski,  Mrs.  Nancy  Evans,  Mr.  Patrick  McKenna,  Captain  Dave  Lyle,  Maj  Mark 
Gallagher  and  Dr.  Jacqueline  Henningsen,  FS. 


USAFA/DFMS),  Mrs.  Andree  D,  New¬ 
man  (AFSAA/CC) 

Junior  Civilian:  Mr.  Stephen  R.  Ganger 
(AFOTEC) 


Please  join  the  Air  Force  Analytic 
Community  in  congratulating  the  award 
winners  and  nominees.  Q 


Barchi  Prize  Paper 

An  Application  of  Exploratory  Analysis: 

The  Weapons  Mix  Problem 

Arthur  Brooks 
The  RAND  Graduate  School 
Bart  Bennett  and  Steve  Bankes 
RAND 

Over  the  last  several  years,  a  new  approach 
to  model-based  analysis  has  been  developed  at 
RAND.  This  approach,  exploratory  analysis, 
greatly  expands  on  traditional  analytic 
approaches  in  order  to  enhance  understanding 
of  complex  problems,  provide  a  wider  range 
of  information  for  decision  makers,  improve 
comparison  between  alternative  models,  and 
thereby  enable  greater  comprehension  of  poli¬ 
cy  options.  This  paper  reviews  the  methodolo¬ 
gy  of  exploratory  analysis  and  its  advantages 
over  traditional  analysis  in  the  context  of  a 
search  for  the  preferred  weapon  mix.  We 
begin  by  walking  through  a  traditional  analytic 


approach  and  showing  the  kinds  of  results  that 
are  often  observed.  We  then  perform 
exploratory  analysis,  requiring  a  large  number 
of  computational  experiments-on  the  same 
problem,  and  show  that  it  provides  more  infor¬ 
mation  and  keener  insights  than  we  originally 
obtained.  We  continue  by  describing 
exploratory  analysis  more  generally,  and 
demonstrate  its  benefits  to  the  decision  maker 
and  the  analyst.  We  also  discuss  what  is 
required  for  its  routine  use. 

Rist  Prize  Paper 

Ibe  Generation,  Use,  and  Misuse  of 
"PICs"  in  Vulnerability/Lethality  Analyses 

Dr.  Paul  H,  Deitz  and  Dr.  Michael  W.  Starks 

Survivability/Lethality  Analysis  Directorate 
Army  Research  Laboratory 

The  Rist  Prize  Paper  will  be  published  in  the 
Military  Operations  Research  ]0\imsX  in  1999. 
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Clayton  Thomas,  FS  and  Thomas  Allen 
Receive  the  1998  Air  Force  Operations 
Research  Lifetime  Achievement  Award 

Maj  Harry  Newton  (USAF  Academy) 

Maj  Robert  Morris  (Air  Force  Studies  and  Analyses  Agency) 


Mr.  Clayton  Thomas,  FS  (SES) 
and  Col  (Ret)  Thomas  Allen 
were  awarded  Lifetime  Achieve¬ 
ment  Awards  by  Dr.  David  Finkleman 
(SES-4)  for  contributions  to  Military  Opera¬ 
tions  Research  at  the  recent  Air  Force  Oper¬ 
ations  Research  Symposium.  The  sympo¬ 
sium  was  held  at  the  Air  Force  Academy 
and  attended  by  about  150  analysts. 

Mr.  Thomas  was  recognized  for  56  years 
of  contributions  of  applying  operations 
research  and  computer  simulation  to  prob¬ 
lems  of  strategic  and  tactical  importance  to 
the  Air  Force  and  the  Department  of 
Defense.  Starting  as  a  member  of  the 
Army  Air  Corps,  Mr.  Thomas’  contribu¬ 
tions  span  the  entire  history  of  the  Air 
Force.  In  particular,  he  helped  shape  the 
Air  Force  approach  to  quantifying  military 
effectiveness  and  was  singularly  responsible 
for  bringing  a  new  generation  of  air  and 
space  systems  into  the  Air  Force.  He  has 
been  a  mentor  for  analysts,  both  inside  and 
outside  the  Air  Force,  for  a  generation.  His 
other  awards  include:  Presidential  Rank  of 
Meritorious  Senior  Executive  Service 
Award,  Air  Force  Exceptional  Civilian  Ser¬ 
vice  Award,  Air  Force  Meritorious  Civilian 
Service  Award,  and  the  Air  Force  Associa¬ 
tion  Citation  of  Honor.  Clayton  Thomas  is 
a  Fellow  of  the  American  Association  for 
the  Advancement  of  Science  and  the  Mili- 


Clayton  Thomas ’s  Lifetime  Achievement 
Award, 


tary  Operations  Research  Society,  which 
recently  established  an  award  in  his  name  to 
honor  individuals  for  consistent,  sustained 
technical  contributions  to  improve  the  ana¬ 
lytical  underpinnings  of  the  military  opera¬ 
tions  research  profession.. 

Colonel  (Ret)  Thomas  Allen,  Ph.D.,  was 
awarded  the  AFORS  Lifetime  Achievement 
award  based  on  his  contributions  over  29 
years  of  service,  as  an  analyst  and  leader. 
An  F-15  pilot  and  Wing  Commander,  he 
helped  found  the  Air  Force  Analytic  Com¬ 
munity  Steering  Group,  which  sponsors 
AFORS  and  is  a  key  reason  for  the  resur¬ 
gence  of  analysis  in  the  Air  Force.  In  his 
last  assignment  as  Commander,  Air  Force 
Studies  and  Analyses  Agency,  Col  Allen 
oversaw  over  200  major  studies  shaping  Air 
Force  weapons  systems  and  force  structure, 
and  brought  the  service  into  full  partnership 
with  the  rest  of  the  military  in  wargaming 
and  distributed  simulation.  He  retired 
effective  1  July  1998  and  is  currently  a 
member  of  the  professional  research  staff  of 
the  Institute  for  Defense  Analyses.  Dr. 
Allen  was  a  MORS  Barchi  Award  winner,  a 
National  Security  Fellow  at  Harvard  Uni¬ 
versity,  and  the  recipient  of  the  Legion  of 
Merit,  with  one  oak  leaf  cluster,  the  Merito¬ 
rious  Service  Award  with  three  oak  leaf 
clusters  and  the  Air  Force  Commendation 
Medal.O 


Thomas  Allen's  Lifetime  Achievement 
Award. 


Awards  and 
Recognition 
Update 

The  Military 
Operations 
Research 
Society  exists  to 
enhance  the  profes¬ 
sion  of  military  oper- 
ations  research. 
Therefore,  MORS 
seeks  to  recognize 
excellence  in  the  pro- 
fession.  Annually 
MORS  presents 
awards  and  prizes  in 
direct  support  of  the  Society’s  purpose  to 
enhance  the  quality  and  effectiveness  of 
military  operations  research  through  the 
recognition  of  outstanding  individuals  in 
the  profession  and  the  accomplishment  of 
outstanding  work  by  individuals  and 
teams. 

Since  1978,  the  Vance  R,  Wanner 
award  has  been  awarded  annually  to  rec¬ 
ognize  outstanding  individuals  for  consis¬ 
tent,  sustained  contributions  and  dedica¬ 
tion  to  the  military  operations  research 
profession.  It  was  named  in  honor  of  the 
first  Executive  Secretary  of  the  Society. 

In  June  1998,  the  MORS  Board 
approved  two  new  awards.  The  first  is  the 
Clayton  J.  Thomas  Award  to  recognize 
outstanding  individuals  for  consistent,  sus¬ 
tained  technical  contributions  to  improve 
the  analytical  underpinnings  of  the  mili¬ 
tary  operations  research  profession.  It  was 
named  after  Clayton  J.  Thomas,  FS,  our 
respected  colleague,  who  has  given,  and 
continues  to  give,  so  much  of  enduring 
value  to  the  military  operations  research 
community. 

The  second  new  award  is  the  John  K. 
Walker,  Jr.  Award  to  recognize  the 
author  of  the  technical  article  judged  to  be 
the  best  published  in  PHALANX  during 
the  previous  calendar  year.  It  was  named 
after  Jack  Walker,  FS,  who  was  Editor  of 
PHALANX  for  12  years  and  Editor  Emeri¬ 
tus  for  seven  years. 

Since  1965,  the  Rist  Prize  has  been 
(5^^  UPDATE, p.  52) 
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awarded  annually  for  the  best  paper  sub¬ 
mitted  in  response  to  a  call  for  papers.  It 
is  named  after  David  Rist,  an  early  direc¬ 
tor.  Since  1983,  the  Barchi  prize  has  been 
awarded  annually  for  the  best  paper  pre¬ 
sented  at  the  annual  Symposium.  It  is 
named  in  honor  of  CDR  Richard  H. 
Barchi,  who  was  a  former  director.  For 
several  years,  MORS  has  awarded  the 
Graduate  Research  Prize  for  the  best  the¬ 
sis  in  each  operations  research  class  at  the 
Air  Force  Institute  of  Technology  and  the 
Naval  Postgraduate  School. 

For  years,  the  famous  MCAP  Award 

—  MORS  Coveted  Acrylic  Paperweight 

—  has  been  given  to  keynote  speakers, 
special  meeting  chairpersons  and  others 
who  we  wanted  to  thank  for  their  unique 
participation  and  support  to  MORS. 

Since  1997,  MORS  board  members 
and  other  senior  members  of  the  commu¬ 
nity  have  been  authorized  to  award  “on 
the  spot”  the  MORS  coin  for  excellence  or 
MORS  “Impact  Award”  for  useful  and 
meaningful  analysis  or  other  actions  that 
have  positive  impact  on  MORS. 

Last,  but  not  least,  is  the  very  impor¬ 
tant,  instant  recognition  by  supervisors 
and  peers  of  superb  analysis  by  a  “pat  on 
the  back”  with  the  words  “well  done.”  O 


BIOMEDICAL  MODELS 

( continued  from  p.  17) 

Summary 

The  concept  of  using  relatively  simple 
first  order  models  as  a  building  block  for 
more  complex  models  is  summarized  in 
Figure  1.  As  discussed  above,  modeling  all 
the  detailed  manifestations  of  the  human 
system  is  analytically  intractable.  This  is 
due  to  many  factors  including  the  large 
number  and  interplay  of  external  and  inter¬ 
nal  variables,  nonlinearity,  time  varying 
nature  of  responses,  nonstationary  system 
properties  and  effects  that  are  difficult  to 
quantitatively  represent.  Initial  and  bound¬ 
ary  conditions  for  dynamic  system  response 
functions  may  also  be  burdened  with  related 
complexities. 

Although  comprehensive  modeling  of 
the  human  system  is  a  significant  analysis 
challenge,  simplification  methods  can  be 
invoked  to  manage  the  complexity  to  create 


MORS  PRESIDENT 

(continued from  p.  3) 

Volunteer  Society  - 
Tribute  to  Bosses 

MORS  relies  on  its  volunteers  to  con¬ 
duct  the  business  not  handled  by  our  won¬ 
derful  MORS  staff.  Not  only  does  this 
require  time  and  effort  fi*om  the  individual, 
during  and  after  normal  working  hours,  but 
also  requires  the  full  support  of  their  boss¬ 
es.  I  would  like  to  take  this  “halftime” 
break  to  personally  thank  my  boss  Dr.  Bill 
Lese,  FS;  who  also  supports  Cy  Staniec, 
mentioned  above  on  the  Board  of  Direc¬ 
tors.  I  would  also  like  to  give  a  standing 
ovation  to  the  bosses  of  other  Board  of 


Directors  members  listed  below. 

Locker  Room  Pep  Talk 

The  MORS  staff  and  the  volunteers  of 
this  Society  have  certainly  put  together 
strong  first  half  statistics  (no  pun  intended 
in  an  OR  crowd).  But  as  everyone  knows, 
it  is  the  score  at  the  end  of  the  ballgame 
that  counts.  We  must  put  together  a  strong 
second  half  to  be  successful.  We  are  also 
looking  at  getting  our  younger  players 
(Junior  Analyst)  some  playing  time  by  par¬ 
ticipation  in  the  Education  Colloquium, 
other  MORS  committees  and  the  67th 
MORSS.  Let’s  rush  out  of  the  locker  room 
and  work  as  a  team.  Show  everyone  the 
MORS  Society  is  an  All-Pro  outfit! 


Board  of  Directors  Member  Boss 


Dr.  Jerry  Kotchka 
Dr.  Roy  Rice 
Dr.  Bob  Sheldon 
Ms.  Sue  Iwanski 
CAPT  Lee  Dick 
Dr.  Tom  Allen 
Dr.  Dean  Hartley 
Ms.  Anne  Patenaude 
Dr.  Yupo  Chan 
Maj  Mark  Gallagher 

Ms.  Mary  Bonnet 
CDR  Kirk  Michealson 
COL  Crash  Konwin 
Mr.  Ted  Smyth 


Mr.  Dave  Swain,  Executive  Vice  President 

Mr.  John  Reece,  VP  for  Defense  Programs 

Mr.  Wayne  Zandbergen,  President 

Mr.  James  Durham,  Vice  President 

VADM  Patricia  Tracey,  Director  of  Naval  Training  (N7) 

Dr.  David  L.  Randall,  Director,  System  Evaluation  Division 

Dr.  Harvey  Gray,  Director 

Dr.  Ralph  Wetzl,  Corporate  Vice  President 

Dr.  Robert  Calico,  Dean 

Col  John  M.  Andrew,  Head  Dept,  of  Organizational 
Sciences/AFTT 

Col  Kurt  A.  Cichowski,  AFSAA  Commander 
Mr.  Thomas  L.  Gibson,  Naval  Forces  Division  Director 
Dr.  Delores  Etter,  Deputy  Director,  Defense  Research  &  Eng 
Russell  E.  Gmgras,  Director,  Joint  Warfare  Analysis  Dept  Q 


manageable  application-specific  models. 
Conversely,  during  synthesis,  basic 
response  models  can  be  enhanced  with 
additional  complexity  to  enhance  their 
capabilities  and  scope  of  application. 
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(continued from  p.  7) 

Specification  of  Four  Games 

Let  the  payoff  matrix  of  the  game  be 
denoted  by: 

A  =  ajj  (i=l,...,m  and  j  =  l,...,n). 

1.  Game  1  (Both  Sides  Have 
Correct  Information) 

In  the  first  game  Side  1  and  Side  2 
have  common  and  correct  knowledge  of 
all  of  the  values  of  the  payoff  matrix  A. 
Side  1  chooses  his  maxmin  strategy  and 
Side  2  chooses  his  minmax  strategy  and 
the  payoff  corresponding  to  this  choice 
results.  The  payoff  of  the  game  is: 

Payoff  of  Game  1  =  aj*j*. 

Since  the  underlying  payoffs  are  ran¬ 
dom  and  distributed  uniformly  between  0 
and  100  the  expected  payoff  is  50.  Any 
other  result  of  the  1000  trials  would  indi¬ 
cate  that  the  computer  program  is  incor¬ 
rect. 

2.  Game  2  (Side  1  Has  Correct 
Information  and  Side  2 
Has  Incorrect  Information) 

In  the  second  game  Side  1  has  correct 
knowledge  of  all  of  the  values  of  A  (call 
this  matrix  Aj)  and  Side  2  has  a  complete¬ 
ly  incorrect  payoff  matrix  A2  composed 
of  a  second  set  of  random  numbers 
between  0  and  100. 

Side  1  chooses  his  maxmin  strategy 
from  the  correct  matrix  Aj.  Side  2  choos¬ 
es  his  minmax  strategy  j*(Game  2)  from 
the  incorrect  matrix  A2.  The  payoff  of  the 
game  is: 

Payoff  of  Game  2  =  ajH=j=i=(Game  2)- 

3.  Game  3  (Side  1  Has  Correct 
Information,  Side  2  Has  Correct 
Information  and  Side  1  Knows 
Side  2’s  Choice) 

In  the  third  game  both  Side  1  and  Side 
2  have  correct  knowledge  of  the  values  of 
A. 

Side  2  chooses  his  minmax  strategy  j* 
from  the  correct  matrix  A. 

Side  1  knows  the  choice  of  Side  2. 
Rather  than  choose  his  maxmin  strategy 
he  focuses  only  on  the  payoffs  corre¬ 


sponding  to  the  minmax  choice  of  Side  2 
and  maximizes  his  payoff.  From  the  cor¬ 
rect  matrix  A  Side  1  chooses: 

i*(Game  3)  =  i  corresponding 
to  maximum  ajj*. 
i 


Side  2  and  Side  I’s  knowing 
Side  2’s  move,  yields  a 
marginal  gain  of  41.2 
over  the  expected  payoff  of  50.0. 

The  payoff  of  the  game  is: 

Payoff  of  Game  3  =  ai*(Game  3)j** 


4.  Game  4  (Side  1  Has  Correct  Informa¬ 
tion,  Side  2  Has  Incorrect  Information 
and  Side  1  knows  Side  2’s  Choice) 

In  the  fourth  game  Side  1  has  correct 
knowledge  of  all  of  the  values  of  A  (call 
this  matrix  Af)  and  Side  2  has  a  complete¬ 
ly  incorrect  payoff  matrix  A2  composed  of 
a  second  set  of  random  numbers  between  0 
and  100. 

Side  2  chooses  his  minmax  strategy 
j*(Game  4)  from  the  incorrect  information 
ofA2. 

Side  1  knows  the  choice  of  Side  2. 
Rather  than  using  his  maxmin  strategy  he 
focuses  only  on  the  payoffs  corresponding 
to  the  minmax  choice  of  Side  2  from  the 
incorrect  information.  From  the  correct 
matrix  A  j  Side  1  chooses: 

i*(Game  4)  =  i  corresponding 

to  maximum  ajj*(G3jne  4)- 

i 

The  payoff  of  the  game  is: 

Payoff  of  Game  4  = 

^*(Game  4)j*(Game  4)- 


Results 

Results  are  given  for  three  sets  of  four  games.  They  are  as  follows: 


Number  of  Strategies 


per  Side 

Game  1 

Game  2 

Game  3 

Game  4 

3X3 

50.0 

62.5 

57.5 

75.2 

5X5 

50.2 

60.8 

65.4 

83.0 

10X10 

48.9 

58.9 

75.4 

91.2 

These  results  show  that  the  expected 
payoff  is  highly  sensitive  to  the  informa¬ 
tion  of  both  sides. 

For  Game  1  the  expected  payoff  is 
always  approximately  50,  which  validates 
the  simulation. 

For  the  10X10  case  the  increase  in 
expected  payoff  from  Game  1  to  Game  2 
to  Game  3  to  Game  4  is  very  interesting. 
Game  2,  with  Side  2  having  bad  informa¬ 
tion,  yields  a  marginal  gain  of  8.9  over  the 
expected  payoff  of  50.0.  Game  3,  with 
Side  1  knowing  Side  2’s  move  (even 
though  Side  2  has  good  information)  yields 
a  marginal  gain  of  25.4  over  the  expected 


payoff  of  50.0.  Game  4,  with  both  bad 
information  for  Side  2  and  Side  I’s  know¬ 
ing  Side  2’s  move,  yields  a  marginal  gain 
of  41 .2  over  the  expected  payoff  of  50.0. 

For  the  3X3  case  the  effects  are  quite 
different  than  for  the  10X10  case.  Game  2, 
where  Side  2  has  bad  information,  yields  a 
marginal  gain  of  12.5  over  the  expected 
payoff  of  50.0,  compared  with  8.9  in  the 
10X10  case.  Game  3,  where  Side  1  knows 
Side  2’s  move,  yields  a  marginal  gain  of 
7.5,  compared  with  25.4  in  the  10X10  case. 
Game  4,  where  Side  2  has  bad  information 
and  Side  1  knows  Side  2’s  move,  yields  a 
(See  GAMES,/?.  34) 
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(continued  from  p,  33) 

marginal  gain  of  25.2,  compared  with  41.2 
in  the  10X10  case. 

The  overall  effects  of  information  are 
not  usually  as  great  in  the  3X3  case  as  in 


■  The  insight  to  be  gleaned 
from  this  analysis  is  that 
non-optional  decision  rules 
can  lead  to  significantly  inferior  resuls 
compared  to  optimal 
game-theoretic  solutions. 

the  10X10  case  but  there  are  some  excep¬ 
tions.  Bad  information  for  Side  2  has  more 
relative  effect  m  the  3X3  case.  Side  Ts 
knowing  Side  2’s  choice  has  more  relative 
effect  in  the  10X10  case.  Combining  the 
effects  of  Side  2  having  bad  information 
and  Side  1  knowing  Side  2’s  choice  has  a 
relatively  more  pronounced  outcome  in  the 
3X3  case  than  in  the  10X10  case.  That  is, 
in  the  3X3  case  the  increase  of  25.2  in 
Game  4  is  much  larger  than  either  of  the 
single  effects  in  Game  2  and  Game  3, 
while  in  the  10X10  case  the  increase  of 
41 .5  in  Game  4  is  not  that  much  larger  than 
the  single  effect  in  Game  3. 

There  is  an  interesting  technical  expla¬ 
nation  for  the  relative  sensitivity  to  infor¬ 
mation  of  the  3X3  case  and  the  10X10  case 
in  Game  2.  In  the  3X3  case  there  is  a  fairly 
high  probability  that  all  three  of  the  values 
for  a  particular  choice  of  Side  1  will  be 
equal  to  or  greater  than  50,  namely  .5^  = 
.125.  The  probability  that  at  least  one  of 
the  three  choices  of  Side  1  will  be  equal  to 
or  greater  than  50  is  1  -  .875^  =  .33  Since 
Side  2’s  choice  is  essentially  random  due 
to  having  bad  information,  Side  1  will 
receive  either  the  expected  value  of  50  or 
will  receive  the  improved  value  about  one- 
third  of  the  time. 

In  the  10X10  case,  however,  there  is  a 
low  probability  that  all  of  the  values  for  a 
particular  choice  of  Side  1  will  be  equal  to 
or  greater  than  50,  namely  .5^^  =  ,00097. 


So  Side  1  can’t  capitalize  on  choosing  a 
strategy  with  a  payoff  greater  than  50 
much  of  the  time.  Instead  Side  1  must  set¬ 
tle  for  the  improvements  in  outcome  due 
solely  to  the  informed  choice  of  Side  1 
coupled  with  the  uninformed  choice  of 
Side  2. 

In  Game  3,  as  compared  with  Game  2, 
there  is  a  different  set  of  driving  factors.  In 
the  3X3  case  Side  2  can  guarantee  that 
Side  1  will  receive  a  certain  fairly  low  pay¬ 
off  and,  even  though  Side  1  knows  Side  2’s 
strategy.  Side  1  cannot  significantly  exploit 
this  knowledge.  In  the  10X10  case,  how¬ 
ever,  Side  2  cannot  guarantee  that  Side  1 
will  receive  a  fairly  low  payoff  and  Side  1 
has  a  wide  range  of  options  for  taking 
advantage  of  the  information  about  Side 
2’s  choice.  Thus  the  outcomes  differ  great¬ 
ly  between  the  3X3  case  and  10X10  case 
in  Game  3. 


There  are  no  significant  differences  in 
Game  2  and  Game  4.  This  is  because  Side 
2  has  incorrect  information  in  these  games 
and  so  non-optimal  decision  rules  are 
equivalent  to  optimal  decision  rules. 

There  are  significant  differences,  how¬ 
ever,  in  Game  1  and  Game  3.  The  adop¬ 
tion  by  Side  2  of  a  non-optimal  decision 
rule  in  the  10X10  case  increases  Side  I’s 
payoff  by  an  increment  of  almost  6  per¬ 
cent  in  Game  1  and  10  percent  in  Game  3. 

The  insight  to  be  gleaned  from  this 
analysis  is  that  non-optimal  decision  rules 
can  lead  to  significantly  inferior  results 


Sensitivity  to  Decision  Rules  Versus 
Game-Theoretic  Solutions 

Results  of  four  more  games  are  present¬ 
ed  here.  These  games  include  a  myopic 
decision  rule  for  Side  2  as  follows.  Side  2 
first  chooses  strategy  1.  He  then  examines 
strategy  2  to  see  if  it  reduces  the  maximum 
which  Side  2  can  lose;  if  so  he  adopts  strat¬ 
egy  2  and  if  not  he  settles  finally  on  strate¬ 
gy  1 .  If  he  adopts  strategy  2  he  then  exam¬ 
ines  strategy  3  to  see  if  it  reduces  the 
maximum  which  Side  2  can  lose;  if  so  he 
adopts  strategy  3  and  if  not  he  settles  final¬ 
ly  on  strategy  2.  This  procedure  is  per¬ 
formed  until  he  settles  finally  on  a  strategy. 

The  above  procedure  is  representative 
of  a  non-optimal  decision  rule  rather  than  a 
game-theoretic  optimal  strategy. 

A  new  set  of  computations  is  performed 
for  the  four  original  games  and  the  four 
new  games.  This  time  the  games  are 
repeated  2000  times  with  a  different  ran¬ 
dom  number  seed. 


compared  to  optimal  game-theoretic  solu¬ 
tions. 
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Results  are  as  follows  for  the  four  original  games: 


Number  of  Strategies 


per  Side 

Game  1 

Game  2 

Game  3 

Game  4 

3X3 

50.1 

61,2 

58.0 

74.5 

5X5 

50.5 

60.5 

65.3 

83.7 

10X10 

49,7 

59.1 

75.0 

90.8 

Results  are  as  follows  for  the  four  new  games  with  Side  2  using 

non-optimal  decision  rules: 

Number  of  Strategies 

per  Side 

Game  1 

Game  2 

Game  3 

Game  4 

3X3 

51.7 

61.9 

60.5 

74.5 

5X5 

54.4 

60.5 

72.8 

83.5 

10X10 

55.6 

57.9 

85.0 

90.7 
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Introduction  to  PHALANX  Online 


C  APT  Lee  Dick 


This  edition  of 
PHALANX 
begins  a  new 
horizon,  a  web  sup¬ 
plement,  PHALANX 
Online.  Two  years 
ago,  as  Electronic 
Media  Chairman,  I 
vowed  to  bring  the 
Society  into  the  elec¬ 
tronic  age,  by  institut¬ 
ing  a  MORS  web  homepage  and  by  lever¬ 
aging  the  power  of  electronic  email  to 
stimulate  activity  and  professional  discus¬ 
sions.  I’ve  now  volunteered  to  assume  the 
role  of  PHALANX  Associate  Editor,  to 
bring  the  Society's  premier  voice,  PHA¬ 
LANX,  into  the  21st  century. 

First,  the  web  edition  will  complement, 
not  replace  or  duplicate,  our  familiar  and 
well  cherished  published  document.  The 
first  online  edition  provided  an  overview  of 
the  key  articles  in  the  upcoming  published 
version.  My  hope  is  that  it  made  you  as 
anxious  to  read  the  articles  as  it  did  for  me 
to  write  these  summaries  in  an  attempt  to 
pique  your  interest.  As  we  expand  the 
online  capability,  we  will  now  have  a  con¬ 
veyance  for  administrative  articles,  and 
announcements  which  often  times  were 
either  omitted  or  abbreviated  because  of  a 


lack  of  space,  or  they  choked  out  the  entree, 
a  technical  article.  As  we  continue  to  grow 
the  online  capability,  we  may  even  want  to 
publish  a  technical  article  in  its  entirety  on 
the  web.  It  would,  of  course,  compete  with 
the  regular  published  articles  as  a  Walker 
Award  candidate.  Any  volunteers?  We  can 
do  multi-media! 

The  online  PHALANX  will  also  greatly 
add  to  the  exchange  of  professional  infor¬ 
mation,  such  as  feedback  or  counterpoint 
articles.  This  is  key  in  stimulating  growth 
and  interest  in  the  Society  yet  our  current 
method  is  relegated  to  a  subsequent  edition 
or  two  of  PHALANX  to  generate  or  com¬ 
ment  on  another  article.  This  is  much  too 
slow.  We  can  leverage  on  the  power  of  the 

web  to  improve  that  process  by  at  least 
an  order  of  magnitude.  My  vision  is  to 
bring  this  online,  with  the  ability  to  insert 
short  paragraphs  in  an  online  discussion 
area,  to  either  add  to  or  debate  a  published 
article. 

In  closing,  I  leave  you  with  one 
thought.. ..I'm  sure  Jack  Walker  would 
have  endorsed  this  wholeheartedly  as  a 
measure  to  significantly  improve  the  profes¬ 
sional  communications  within  our  Society. 

May  the  final  year  of  the  millennium 
bring  you  much  happiness  and  success!  O 


Lee  Dick 


Simulation  Validation  (SEVTVAL)  Workshop  ’99 

Making  W&A  Effective  AND  Affordable 
26-28  January  1999,  Johns  Hopkins  University/APL 


In  the  late  1980s,  MORS  initiated  a 
series  of  workshops  to  serve  as  a  forum 
for  dialogue  on  the  issue  of  simulation 
validation  (SIMVAL).  That  forum  was  the 
genesis  of  DoD’s  definitions  for  Verifica¬ 
tion,  Validation  and  Accreditation 
(VV&A),  which  have  become  central 
themes  in  DoD  modeling  and  simulation. 
Recent  developments  in  the  policies  and 
procedures  governing  W&A  underscore  a 
new  need  to  advance  the  state-of-the-art  in 
VV&A  technologies  and  relate  those  tech¬ 
nologies  to  the  needs  of  DoD  organizations. 
This  examination  should  include  not  only 
tools  and  methodologies  for  performing 
V&V,  but  should  also  address  costs  associ¬ 
ated  with  conducting  W&A.  In  particular, 


technologies  should  be  identified  that  can 
increase  V&V  effectiveness  without 
increasing  costs,  or  decrease  costs  without 
degrading  V&V  effectiveness. 

To  address  this  need  the  Military  Opera¬ 
tions  Research  Society  has  organized  a 
Workshop  to  examine  the  question  of 
"Making  VV&A  Effective  AND  Afford¬ 
able."  This  Workshop  will  be  held  26-29 
January  1999  at  the  Kossiakoff  Conference 
Center  of  Johns  Hopkins  University/ 
Applied  Physics  Laboratory,  in  Laurel, 
Maryland.  If  you  would  like  more  informa¬ 
tion  about  this  meeting  or  would  like  an 
application  to  attend  please  visit  our  web 
site  at  www.mors.org  or  call  the  MORS 
office  at  (703)  751-7290.0 


TESTING 

(continued  from  p.  15) 

focused  on  priority  user  concerns. 

•  Test  planning  and  reporting  was  facilitated 
because  one  measure  of  performance 
applied  to  three  critical  operational  issues. 

•  The  validity  of  the  results  was  enhanced 
because  SMEs  from  the  user  community 
were  the  best-qualified  persons  to  assess 
mission  tasks. 

Summary,  Application  of  this  measure 
of  performance  requires  a  significant  com¬ 
mitment  from  the  user  community.  For 
example,  the  users  provide  input  to  the  mis¬ 
sion  task  list  and  provide  the  SMEs.  The 
result  is  a  test  highly  focused  on  user  con¬ 
cerns. 

In  any  system  where  the  requirements 
are  ‘"soft”  and  there  is  a  lack  of  quantitative 
thresholds,  this  focus  on  the  ability  of  the 
system  to  support  user  missions  facilitates 
the  operational  evaluation.  The  technique 
of  assessing  task  success  based  on  the  good¬ 
ness  of  information  can  be  applied  to  other 
C4I  systems.  The  characteristics  of  timeli¬ 
ness,  accuracy,  completeness  and  usefulness 
are  appropriate  measures  for  information 
goodness. 

The  views  expressed  in  this  article  are 
those  of  the  author(s)  and  do  not  reflect  the 
official  policy  or  position  of  the  Department 
of  Defense  or  the  US  Government. 

Biographies 

Lieutenant  Commander  Mark  Thomp¬ 
son  is  the  JITC  GCCS  operational  test 
director.  He  is  a  graduate  of  the  US  Naval 
Academy  and  received  his  Master  of  Arts  in 
National  Security  and  Strategic  Studies 
from  the  Naval  War  College.  He  has  13 
years  experience  as  a  Navy  submariner. 

Dr.  Ernest  Montague  is  the  Chief  Sci¬ 
entist  for  Systems  Test,  Evaluation  and 
Analysis  at  BDM  International.  He 
received  his  Doctor  of  Science  in  Opera¬ 
tions  Research  from  George  Washington 
University.  He  has  over  30  years  profes¬ 
sional  experience,  to  include  10  years  in 
OT  of  major  Defense  systems.  He  is  a 
member  of  ITEA,  the  Military  Operations 
Research  Society  (MORS),  the  Institute  for 
Operations  Research  and  Management  Sci¬ 
ence  (INFORMS),  and  the  Armed  Forces 
Communications  Electronics  Association 
(AFCEA).  O 
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PHALANX  Online 

Julian  Palmore 

The  use  of  electronic  media  by  LANX  for  PHALANX  Online.  This  step  is  With  Lee's  enthusiasm  and  extensive 

MORS  has  increased  and  matured  significant.  As  an  electronic  supplement  knowledge  about  the  use  of  the  web  and 

over  the  past  several  years.  It  is  PHALANX  Onhne  will  allow  PHALANX  the  conversion  of  MORS  activities  to  it 

now  time  for  PHALANX  to  take  a  step  to  reach  a  wider  audience  more  effective-  I'm  sure  PHALANX  Online  will  be  a  great 

into  the  electronic  Web  World.  We  do  so  ly.  It  will  allow  larger  versions  of  articles  success. 

at  the  suggestion  of  CAPT  Lee  Dick  who  and  announcements  to  appear  in  full  as  Please  see  Lee  Dick’s  article  on  page 

has  agreed  to  be  Associate  Editor  of  PHA-  supplements  to  that  printed  in  PHALANX.  35  for  more  details.  O 

This  Just  In ... 

Three  new  Fellows  of  the  Society  the  Society  is  reserved  for  individuals  who  Congratulations  are  also  in  order  for 

were  elected  at  the  8  December  have  made  significant  and  lasting  contri-  LTC  Michael  L.  McGinnis.  He  has 

Board  of  Director’s  Meeting.  butions  to  the  Society.  We  thank  these  recently  been  selected  as  Department  head 

They  are,  in  alphabetical  order:  Mr.  Brian  individuals  for  their  continued  service  and  United  States  Military  Academy’s 

McEnany;  Dr.  Stu  Starr;  and  Dr.  Harry  look  forward  to  many  more  productive  Department  of  Systems  Engineering.  O 

Thie.  Elevation  to  the  status  of  Fellow  of  years  with  MORS.  Congratulations!! 
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